2024-04-20T15:41:36.098 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-20T15:41:36.098 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --description rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} --archive /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647 --name yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/orig.config.yaml 2024-04-20T15:41:36.137 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T15:41:36.229 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647 branch: wip-yuri4-testing-2024-04-19-0708-quincy description: rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} email: yweinste@redhat.com first_in_suite: false job_id: '7664647' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-04-19-0708-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph global: osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 7f4fa80c8c97f53f52b1189d2347b5761f6287e3 wait-for-scrub: false ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7f4fa80c8c97f53f52b1189d2347b5761f6287e3 rgw: frontend: beast storage classes: LUKEWARM, FROZEN selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-04-19-0708-quincy sha1: 7f4fa80c8c97f53f52b1189d2347b5761f6287e3 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 367 sha1: 7f4fa80c8c97f53f52b1189d2347b5761f6287e3 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: wip-yuri4-testing-2024-04-19-0708-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7f4fa80c8c97f53f52b1189d2347b5761f6287e3 targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDb2dPrGNonFiPBD42oeSD+ltZCSt7KxYjqQLgxirOxBqzNetAzxLREfOYmzF4MpHrzNQitBAOmQXxMU0UXxK+E= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJr7BmfmWrdB1CO79XiqR7yDTi5evHE7CHFwOfOPXCu5jkFmr+QPbFz8h5Sn5L62sRrU2MLudH+ncq8K5mrgzA= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - rabbitmq: client.0: null - notification-tests: client.0: extra_attr: - amqp_test rgw_server: client.0 - kafka: client.0: kafka_version: 2.6.0 - notification-tests: client.0: extra_attr: - kafka_test rgw_server: client.0 - notification-tests: client.0: rgw_server: client.0 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-20_15:31:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-20T15:41:36.229 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa; will attempt to use it 2024-04-20T15:41:36.230 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks 2024-04-20T15:41:36.230 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-20T15:41:36.232 INFO:teuthology.task.internal:Checking packages... 2024-04-20T15:41:36.257 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7f4fa80c8c97f53f52b1189d2347b5761f6287e3' 2024-04-20T15:41:36.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T15:41:36.257 INFO:teuthology.packaging:ref: None 2024-04-20T15:41:36.257 INFO:teuthology.packaging:tag: None 2024-04-20T15:41:36.257 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-04-19-0708-quincy 2024-04-20T15:41:36.257 INFO:teuthology.packaging:sha1: 7f4fa80c8c97f53f52b1189d2347b5761f6287e3 2024-04-20T15:41:36.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-04-19-0708-quincy 2024-04-20T15:41:36.468 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-902.g7f4fa80c 2024-04-20T15:41:36.469 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-20T15:41:36.475 INFO:teuthology.task.internal:no buildpackages task found 2024-04-20T15:41:36.475 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-20T15:41:36.493 INFO:teuthology.task.internal:Saving configuration 2024-04-20T15:41:36.507 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-20T15:41:36.514 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T15:41:36.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 15:32:48.014844', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDb2dPrGNonFiPBD42oeSD+ltZCSt7KxYjqQLgxirOxBqzNetAzxLREfOYmzF4MpHrzNQitBAOmQXxMU0UXxK+E='} 2024-04-20T15:41:36.558 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 15:32:48.013482', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJr7BmfmWrdB1CO79XiqR7yDTi5evHE7CHFwOfOPXCu5jkFmr+QPbFz8h5Sn5L62sRrU2MLudH+ncq8K5mrgzA='} 2024-04-20T15:41:36.558 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-20T15:41:36.565 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-20T15:41:36.565 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-20T15:41:36.565 INFO:teuthology.run_tasks:Running task console_log... 2024-04-20T15:41:36.651 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f06fb7f70>, signals=[15]) 2024-04-20T15:41:36.652 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-20T15:41:36.658 INFO:teuthology.task.internal:Opening connections... 2024-04-20T15:41:36.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T15:41:36.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:41:36.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T15:41:36.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:41:36.816 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-20T15:41:36.825 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-04-20T15:41:36.847 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-04-20T15:41:36.848 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-04-20T15:41:36.909 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-04-20T15:41:36.909 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T15:41:36.910 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T15:41:36.911 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-04-20T15:41:36.935 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-04-20T15:41:36.958 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-04-20T15:41:36.958 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-04-20T15:41:37.020 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T15:41:37.021 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T15:41:37.021 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T15:41:37.021 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-04-20T15:41:37.044 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-20T15:41:37.055 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-20T15:41:37.067 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-20T15:41:37.067 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-04-20T15:41:37.070 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-04-20T15:41:37.089 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-20T15:41:37.101 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-20T15:41:37.101 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-04-20T15:41:37.131 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-04-20T15:41:37.161 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-20T15:41:37.222 INFO:teuthology.run_tasks:Running task kernel... 2024-04-20T15:41:37.240 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-20T15:41:37.241 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-20T15:41:37.241 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T15:41:37.241 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T15:41:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T15:41:37.262 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-04-20T15:41:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T15:41:37.263 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-04-20T15:41:37.322 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-489.el8.x86_64 2024-04-20T15:41:37.322 INFO:teuthology.task.kernel:Running kernel on smithi119: 4.18.0-489.el8.x86_64 2024-04-20T15:41:37.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-04-20T15:41:37.324 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-489.el8.x86_64 2024-04-20T15:41:37.324 INFO:teuthology.task.kernel:Running kernel on smithi071: 4.18.0-489.el8.x86_64 2024-04-20T15:41:37.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-04-20T15:41:40.984 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - AppStream 27 MB/s | 28 MB 00:01 2024-04-20T15:41:41.137 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - AppStream 24 MB/s | 28 MB 00:01 2024-04-20T15:41:47.638 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - BaseOS 6.6 MB/s | 10 MB 00:01 2024-04-20T15:41:48.142 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - BaseOS 4.9 MB/s | 10 MB 00:02 2024-04-20T15:41:49.855 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-04-20T15:41:50.189 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 7.7 kB 00:00 2024-04-20T15:41:50.284 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-04-20T15:41:50.457 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-20T15:41:50.743 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-04-20T15:41:50.995 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-20T15:41:51.425 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-04-20T15:41:52.629 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-20T15:41:55.216 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 377 kB/s | 24 kB 00:00 2024-04-20T15:41:56.445 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 415 kB/s | 24 kB 00:00 2024-04-20T15:41:58.290 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-20T15:41:58.291 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-20T15:41:58.399 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:41:58.400 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:41:58.400 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-04-20T15:41:58.400 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:41:58.400 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:41:58.400 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout:Install 3 Packages 2024-04-20T15:41:58.401 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:41:58.402 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 90 M 2024-04-20T15:41:58.402 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 96 M 2024-04-20T15:41:58.402 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:41:59.534 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-20T15:41:59.534 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-20T15:41:59.637 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-20T15:41:59.638 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:41:59.639 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:41:59.639 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:41:59.639 INFO:teuthology.orchestra.run.smithi071.stdout:Install 3 Packages 2024-04-20T15:41:59.639 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:41:59.639 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 90 M 2024-04-20T15:41:59.639 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 96 M 2024-04-20T15:41:59.640 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:42:07.025 INFO:teuthology.orchestra.run.smithi119.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 1.2 MB/s | 10 MB 00:08 2024-04-20T15:42:09.033 INFO:teuthology.orchestra.run.smithi119.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.5 MB/s | 36 MB 00:10 2024-04-20T15:42:09.895 INFO:teuthology.orchestra.run.smithi071.stdout:(1/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 3.6 MB/s | 36 MB 00:10 2024-04-20T15:42:10.020 INFO:teuthology.orchestra.run.smithi071.stdout:(2/3): kernel-4.18.0-552.el8.x86_64.rpm 1.0 MB/s | 10 MB 00:10 2024-04-20T15:42:10.684 INFO:teuthology.orchestra.run.smithi119.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 3.6 MB/s | 44 MB 00:12 2024-04-20T15:42:10.684 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:42:10.684 INFO:teuthology.orchestra.run.smithi119.stdout:Total 7.4 MB/s | 90 MB 00:12 2024-04-20T15:42:11.287 INFO:teuthology.orchestra.run.smithi071.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 3.8 MB/s | 44 MB 00:11 2024-04-20T15:42:11.287 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:42:11.288 INFO:teuthology.orchestra.run.smithi071.stdout:Total 7.8 MB/s | 90 MB 00:11 2024-04-20T15:42:11.370 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:42:11.800 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:42:11.800 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:42:11.973 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:42:12.393 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:42:12.393 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:42:13.038 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:42:13.073 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:42:13.642 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:42:15.868 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:42:16.406 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T15:42:16.490 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:42:17.136 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T15:42:20.036 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T15:42:20.431 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T15:42:20.677 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T15:42:21.060 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T15:42:23.877 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T15:42:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T15:42:24.792 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:45.951 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:45.975 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:46.315 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:46.315 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-20T15:42:46.315 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-20T15:42:46.681 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:46.705 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:47.049 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:47.049 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-20T15:42:47.049 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-20T15:42:47.307 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:47.307 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:42:47.307 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:42:47.307 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.307 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.307 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:42:47.308 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:42:47.468 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:42:47.577 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:42:47.686 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.686 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T15:42:47.686 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-20T15:42:47.686 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-20T15:42:47.686 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-20T15:42:47.687 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-20T15:42:47.687 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-04-20T15:42:47.687 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T15:42:47.730 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T15:42:47.898 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.899 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T15:42:47.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-04-20T15:42:47.932 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-04-20T15:42:47.932 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-04-20T15:42:47.932 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.02 2024-04-20T15:42:47.932 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 129.el8 2024-04-20T15:42:47.932 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-04-20T15:42:47.932 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Group : System Environment/Base 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 9265586 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Relocations : (not relocatable) 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : CentOS BuildSystem 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T15:42:47.933 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:42:47.934 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T15:42:47.935 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T15:42:47.935 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 4.18.0-552.el8.x86_64 2024-04-20T15:42:47.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T15:42:47.949 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T15:42:47.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T15:42:47.949 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-20T15:42:47.949 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-20T15:42:47.949 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-20T15:42:47.950 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-20T15:42:47.950 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-04-20T15:42:47.950 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T15:42:48.160 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T15:42:48.161 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T15:42:48.161 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-04-20T15:42:48.194 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-04-20T15:42:48.194 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-04-20T15:42:48.194 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.02 2024-04-20T15:42:48.194 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 129.el8 2024-04-20T15:42:48.194 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-04-20T15:42:48.194 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Group : System Environment/Base 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 9265586 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Relocations : (not relocatable) 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : CentOS BuildSystem 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:42:48.195 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T15:42:48.196 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T15:42:48.196 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T15:42:48.196 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-04-20T15:42:48.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:42:48.196 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T15:42:48.197 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T15:42:48.197 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 4.18.0-552.el8.x86_64 2024-04-20T15:42:48.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T15:42:49.381 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-04-20T15:42:50.151 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-04-20T15:42:51.963 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-04-20T15:42:51.965 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-04-20T15:42:51.987 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.bP3VWQA9QA 2024-04-20T15:42:51.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bP3VWQA9QA 2024-04-20T15:42:52.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.bP3VWQA9QA 2024-04-20T15:42:52.200 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.bP3VWQA9QA is 6KB 2024-04-20T15:42:52.248 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.bP3VWQA9QA 2024-04-20T15:42:52.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T15:42:52.331 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-20T15:42:52.331 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-20T15:42:52.331 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-20T15:42:52.332 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-20T15:42:52.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-20T15:42:52.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-04-20T15:42:52.736 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-04-20T15:42:52.738 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-20T15:42:52.770 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.FtWuvxuktY 2024-04-20T15:42:52.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FtWuvxuktY 2024-04-20T15:42:52.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.FtWuvxuktY 2024-04-20T15:42:52.982 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.FtWuvxuktY is 6KB 2024-04-20T15:42:53.031 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.FtWuvxuktY 2024-04-20T15:42:53.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T15:42:53.120 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-20T15:42:53.120 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-20T15:42:53.120 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-20T15:42:53.121 INFO:teuthology.orchestra.run.smithi071.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-20T15:42:53.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-20T15:42:53.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-04-20T15:43:22.511 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-20T15:43:22.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-20T15:43:22.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:43:23.282 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-20T15:43:23.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-20T15:43:23.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:43:56.959 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-04-20T15:44:05.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-20T15:44:05.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:44:23.302 DEBUG:teuthology.orchestra.remote:timed out 2024-04-20T15:44:24.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-04-20T15:44:32.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-20T15:44:32.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:44:32.680 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-20T15:44:32.977 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-20T15:44:32.977 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-20T15:44:32.978 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-04-20T15:44:32.994 INFO:teuthology.orchestra.run.smithi071.stdout:4.18.0-552.el8.x86_64 2024-04-20T15:44:32.994 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-20T15:44:32.995 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T15:44:32.995 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T15:44:33.996 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-20T15:44:33.996 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T15:44:34.073 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-04-20T15:44:34.083 DEBUG:teuthology.parallel:result is None 2024-04-20T15:44:36.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-20T15:44:36.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:44:36.755 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-04-20T15:44:37.039 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-20T15:44:37.040 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-20T15:44:37.040 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-04-20T15:44:37.056 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-552.el8.x86_64 2024-04-20T15:44:37.056 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-20T15:44:37.056 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T15:44:37.057 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T15:44:38.057 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-20T15:44:38.058 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T15:44:38.145 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-04-20T15:44:38.164 DEBUG:teuthology.parallel:result is None 2024-04-20T15:44:38.164 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-20T15:44:38.171 INFO:teuthology.task.internal:Creating test directory... 2024-04-20T15:44:38.171 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T15:44:38.218 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T15:44:38.610 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-20T15:44:38.617 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-20T15:44:38.625 INFO:teuthology.task.internal:Creating archive directory... 2024-04-20T15:44:38.625 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T15:44:38.674 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T15:44:39.100 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-20T15:44:39.107 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-20T15:44:39.108 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T15:44:39.110 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T15:44:39.144 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T15:44:39.154 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T15:44:39.520 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T15:44:39.624 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T15:44:39.638 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-20T15:44:39.646 INFO:teuthology.task.internal:Configuring sudo... 2024-04-20T15:44:39.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T15:44:39.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T15:44:40.270 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-20T15:44:40.280 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-20T15:44:40.280 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T15:44:40.282 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T15:44:40.475 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T15:44:40.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T15:44:40.587 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T15:44:40.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T15:44:40.677 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:44:40.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T15:44:40.743 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T15:44:40.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T15:44:40.959 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T15:44:41.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T15:44:41.046 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:44:41.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T15:44:41.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-04-20T15:44:41.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-04-20T15:44:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T15:44:41.207 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T15:44:41.571 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-20T15:44:41.581 INFO:teuthology.task.internal:Starting timer... 2024-04-20T15:44:41.581 INFO:teuthology.run_tasks:Running task pcp... 2024-04-20T15:44:41.591 INFO:teuthology.run_tasks:Running task selinux... 2024-04-20T15:44:41.601 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-20T15:44:41.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-04-20T15:44:41.691 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-04-20T15:44:41.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-04-20T15:44:41.774 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-04-20T15:44:41.775 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-20T15:44:41.775 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-04-20T15:44:41.803 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-04-20T15:44:41.803 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-04-20T15:44:41.836 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-04-20T15:44:41.837 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive'} 2024-04-20T15:44:41.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T15:44:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T15:44:41.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T15:44:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T15:44:41.903 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-20T15:44:41.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-04-20T15:44:41.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-04-20T15:44:41.970 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-20T15:44:41.981 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-20T15:44:41.987 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-20T15:44:42.111 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-20T15:44:42.129 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-20T15:44:42.130 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-20T15:51:17.212 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2024-04-20T15:51:17.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-20T15:51:17.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:51:17.294 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-20T15:51:17.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-20T15:51:17.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-20T15:51:17.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T15:51:17.405 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-04-20T15:51:17.431 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-20T15:51:17.432 INFO:teuthology.run_tasks:Running task clock... 2024-04-20T15:51:17.443 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-20T15:51:17.443 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T15:51:17.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T15:51:17.447 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T15:51:17.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T15:51:17.477 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T15:51:17.491 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T15:51:17.504 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T15:51:17.518 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T15:51:17.562 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-04-20T15:51:17.573 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-04-20T15:51:17.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-04-20T15:51:17.586 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T15:51:17.590 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-04-20T15:51:17.599 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T15:51:17.604 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T15:51:17.618 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T15:51:17.665 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2024-04-20T15:51:17.667 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T15:51:17.667 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-04-20T15:51:17.667 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.667 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.667 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.668 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.683 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2024-04-20T15:51:17.687 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T15:51:17.687 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-20T15:51:17.687 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.687 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.687 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.687 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T15:51:17.688 INFO:teuthology.run_tasks:Running task install... 2024-04-20T15:51:17.697 DEBUG:teuthology.task.install:project ceph 2024-04-20T15:51:17.697 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3'}} 2024-04-20T15:51:17.697 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3'} 2024-04-20T15:51:17.697 INFO:teuthology.task.install:Using flavor: default 2024-04-20T15:51:17.703 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-20T15:51:17.703 INFO:teuthology.task.install:extra packages: [] 2024-04-20T15:51:17.704 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3', 'tag': None, 'wait_for_package': False} 2024-04-20T15:51:17.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7f4fa80c8c97f53f52b1189d2347b5761f6287e3 2024-04-20T15:51:17.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3', 'tag': None, 'wait_for_package': False} 2024-04-20T15:51:17.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7f4fa80c8c97f53f52b1189d2347b5761f6287e3 2024-04-20T15:51:17.847 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/ 2024-04-20T15:51:17.848 INFO:teuthology.task.install.rpm:Package version is 17.2.7-902.g7f4fa80c 2024-04-20T15:51:17.924 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/ 2024-04-20T15:51:17.924 INFO:teuthology.task.install.rpm:Package version is 17.2.7-902.g7f4fa80c 2024-04-20T15:51:18.008 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T15:51:18.008 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:51:18.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T15:51:18.037 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-20T15:51:18.037 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T15:51:18.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T15:51:18.131 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-19-0708-quincy/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T15:51:18.131 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:51:18.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T15:51:18.161 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-20T15:51:18.161 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f4fa80c8c97f53f52b1189d2347b5761f6287e3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T15:51:18.184 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T15:51:18.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T15:51:18.250 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-04-20T15:51:18.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-04-20T15:51:18.309 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T15:51:18.376 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-04-20T15:51:18.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-04-20T15:51:18.514 INFO:teuthology.orchestra.run.smithi119.stdout:78 files removed 2024-04-20T15:51:18.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-04-20T15:51:18.641 INFO:teuthology.orchestra.run.smithi071.stdout:78 files removed 2024-04-20T15:51:18.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-04-20T15:51:19.306 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 215 kB/s | 78 kB 00:00 2024-04-20T15:51:19.367 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 263 kB/s | 78 kB 00:00 2024-04-20T15:51:19.680 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-20T15:51:19.734 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-20T15:51:19.975 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 5.5 kB/s | 1.5 kB 00:00 2024-04-20T15:51:20.085 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2024-04-20T15:51:21.210 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - AppStream 26 MB/s | 28 MB 00:01 2024-04-20T15:51:21.225 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - AppStream 23 MB/s | 28 MB 00:01 2024-04-20T15:51:27.800 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - BaseOS 6.5 MB/s | 10 MB 00:01 2024-04-20T15:51:29.999 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-04-20T15:51:30.468 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 7.7 kB 00:00 2024-04-20T15:51:30.959 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - BaseOS 2.2 MB/s | 10 MB 00:04 2024-04-20T15:51:31.394 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - PowerTools 5.9 MB/s | 5.0 MB 00:00 2024-04-20T15:51:32.604 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-04-20T15:51:33.118 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-04-20T15:51:33.452 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 7.7 kB 00:00 2024-04-20T15:51:33.796 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-04-20T15:51:34.311 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 8 - PowerTools 6.4 MB/s | 5.0 MB 00:00 2024-04-20T15:51:35.363 INFO:teuthology.orchestra.run.smithi071.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-20T15:51:37.631 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 445 kB/s | 24 kB 00:00 2024-04-20T15:51:38.723 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 16 MB 00:03 2024-04-20T15:51:41.335 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:51:41.337 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:51:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 17 M 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 3.5 M 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 3.6 M 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 5.3 M 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 24 M 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 24 k 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T15:51:41.338 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 798 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 505 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 5.6 M 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 45 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 117 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 190 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 371 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 358 k 2024-04-20T15:51:41.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 107 k 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout:Install 21 Packages 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-04-20T15:51:41.340 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:51:41.341 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 64 M 2024-04-20T15:51:41.341 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:51:42.522 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 435 kB/s | 24 kB 00:00 2024-04-20T15:51:43.656 INFO:teuthology.orchestra.run.smithi119.stdout:(1/23): ceph-base-17.2.7-902.g7f4fa80c.el8.x86_ 2.8 MB/s | 5.3 MB 00:01 2024-04-20T15:51:43.757 INFO:teuthology.orchestra.run.smithi119.stdout:(2/23): ceph-selinux-17.2.7-902.g7f4fa80c.el8.x 244 kB/s | 24 kB 00:00 2024-04-20T15:51:43.965 INFO:teuthology.orchestra.run.smithi119.stdout:(3/23): libcephfs2-17.2.7-902.g7f4fa80c.el8.x86 3.7 MB/s | 798 kB 00:00 2024-04-20T15:51:44.207 INFO:teuthology.orchestra.run.smithi119.stdout:(4/23): ceph-radosgw-17.2.7-902.g7f4fa80c.el8.x 7.0 MB/s | 17 MB 00:02 2024-04-20T15:51:44.249 INFO:teuthology.orchestra.run.smithi119.stdout:(5/23): libradosstriper1-17.2.7-902.g7f4fa80c.e 1.7 MB/s | 505 kB 00:00 2024-04-20T15:51:44.366 INFO:teuthology.orchestra.run.smithi119.stdout:(6/23): python3-ceph-argparse-17.2.7-902.g7f4fa 383 kB/s | 45 kB 00:00 2024-04-20T15:51:44.466 INFO:teuthology.orchestra.run.smithi119.stdout:(7/23): python3-ceph-common-17.2.7-902.g7f4fa80 1.1 MB/s | 117 kB 00:00 2024-04-20T15:51:44.583 INFO:teuthology.orchestra.run.smithi119.stdout:(8/23): python3-cephfs-17.2.7-902.g7f4fa80c.el8 1.6 MB/s | 190 kB 00:00 2024-04-20T15:51:44.709 INFO:teuthology.orchestra.run.smithi119.stdout:(9/23): python3-rados-17.2.7-902.g7f4fa80c.el8. 2.9 MB/s | 371 kB 00:00 2024-04-20T15:51:44.926 INFO:teuthology.orchestra.run.smithi119.stdout:(10/23): librgw2-17.2.7-902.g7f4fa80c.el8.x86_6 7.8 MB/s | 5.6 MB 00:00 2024-04-20T15:51:44.992 INFO:teuthology.orchestra.run.smithi119.stdout:(11/23): python3-rgw-17.2.7-902.g7f4fa80c.el8.x 1.6 MB/s | 107 kB 00:00 2024-04-20T15:51:45.185 INFO:teuthology.orchestra.run.smithi119.stdout:(12/23): python3-rbd-17.2.7-902.g7f4fa80c.el8.x 753 kB/s | 358 kB 00:00 2024-04-20T15:51:45.260 INFO:teuthology.orchestra.run.smithi119.stdout:(13/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.5 MB/s | 677 kB 00:00 2024-04-20T15:51:45.310 INFO:teuthology.orchestra.run.smithi119.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 884 kB/s | 44 kB 00:00 2024-04-20T15:51:45.544 INFO:teuthology.orchestra.run.smithi119.stdout:(15/23): ceph-common-17.2.7-902.g7f4fa80c.el8.x 6.3 MB/s | 24 MB 00:03 2024-04-20T15:51:45.569 INFO:teuthology.orchestra.run.smithi119.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 676 kB/s | 259 kB 00:00 2024-04-20T15:51:45.669 INFO:teuthology.orchestra.run.smithi119.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 131 kB/s | 47 kB 00:00 2024-04-20T15:51:45.778 INFO:teuthology.orchestra.run.smithi119.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2024-04-20T15:51:45.853 INFO:teuthology.orchestra.run.smithi119.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 992 kB/s | 306 kB 00:00 2024-04-20T15:51:45.878 INFO:teuthology.orchestra.run.smithi119.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 359 kB/s | 75 kB 00:00 2024-04-20T15:51:46.079 INFO:teuthology.orchestra.run.smithi119.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 5.8 MB/s | 1.7 MB 00:00 2024-04-20T15:51:46.139 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:51:46.141 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:51:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:51:46.141 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:51:46.141 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:51:46.141 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 17 M 2024-04-20T15:51:46.141 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 3.5 M 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 3.6 M 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 5.3 M 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 24 M 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 24 k 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 798 k 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 505 k 2024-04-20T15:51:46.142 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 5.6 M 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 45 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 117 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 190 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 371 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 358 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 107 k 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:51:46.143 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:51:46.144 INFO:teuthology.orchestra.run.smithi071.stdout:Install 21 Packages 2024-04-20T15:51:46.144 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-04-20T15:51:46.144 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:51:46.144 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 64 M 2024-04-20T15:51:46.144 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:51:46.221 INFO:teuthology.orchestra.run.smithi119.stdout:(22/23): librbd1-17.2.7-902.g7f4fa80c.el8.x86_6 10 MB/s | 3.6 MB 00:00 2024-04-20T15:51:46.554 INFO:teuthology.orchestra.run.smithi119.stdout:(23/23): librados2-17.2.7-902.g7f4fa80c.el8.x86 4.9 MB/s | 3.5 MB 00:00 2024-04-20T15:51:46.556 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:51:46.556 INFO:teuthology.orchestra.run.smithi119.stdout:Total 12 MB/s | 64 MB 00:05 2024-04-20T15:51:46.599 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:51:46.645 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:51:46.645 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:51:47.005 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:51:47.008 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:51:47.415 INFO:teuthology.orchestra.run.smithi071.stdout:(1/23): ceph-base-17.2.7-902.g7f4fa80c.el8.x86_ 7.7 MB/s | 5.3 MB 00:00 2024-04-20T15:51:47.473 INFO:teuthology.orchestra.run.smithi071.stdout:(2/23): ceph-selinux-17.2.7-902.g7f4fa80c.el8.x 419 kB/s | 24 kB 00:00 2024-04-20T15:51:47.616 INFO:teuthology.orchestra.run.smithi071.stdout:(3/23): libcephfs2-17.2.7-902.g7f4fa80c.el8.x86 5.5 MB/s | 798 kB 00:00 2024-04-20T15:51:47.708 INFO:teuthology.orchestra.run.smithi071.stdout:(4/23): libradosstriper1-17.2.7-902.g7f4fa80c.e 5.4 MB/s | 505 kB 00:00 2024-04-20T15:51:48.028 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:51:48.236 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-20T15:51:48.324 INFO:teuthology.orchestra.run.smithi071.stdout:(5/23): librgw2-17.2.7-902.g7f4fa80c.el8.x86_64 9.0 MB/s | 5.6 MB 00:00 2024-04-20T15:51:48.383 INFO:teuthology.orchestra.run.smithi071.stdout:(6/23): python3-ceph-argparse-17.2.7-902.g7f4fa 769 kB/s | 45 kB 00:00 2024-04-20T15:51:48.434 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T15:51:48.450 INFO:teuthology.orchestra.run.smithi071.stdout:(7/23): python3-ceph-common-17.2.7-902.g7f4fa80 1.7 MB/s | 117 kB 00:00 2024-04-20T15:51:48.709 INFO:teuthology.orchestra.run.smithi071.stdout:(8/23): ceph-common-17.2.7-902.g7f4fa80c.el8.x8 12 MB/s | 24 MB 00:01 2024-04-20T15:51:48.734 INFO:teuthology.orchestra.run.smithi071.stdout:(9/23): python3-cephfs-17.2.7-902.g7f4fa80c.el8 670 kB/s | 190 kB 00:00 2024-04-20T15:51:48.759 INFO:teuthology.orchestra.run.smithi071.stdout:(10/23): python3-rados-17.2.7-902.g7f4fa80c.el8 7.3 MB/s | 371 kB 00:00 2024-04-20T15:51:48.802 INFO:teuthology.orchestra.run.smithi071.stdout:(11/23): python3-rbd-17.2.7-902.g7f4fa80c.el8.x 5.2 MB/s | 358 kB 00:00 2024-04-20T15:51:48.812 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T15:51:48.826 INFO:teuthology.orchestra.run.smithi071.stdout:(12/23): python3-rgw-17.2.7-902.g7f4fa80c.el8.x 1.6 MB/s | 107 kB 00:00 2024-04-20T15:51:49.018 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/25 2024-04-20T15:51:49.086 INFO:teuthology.orchestra.run.smithi071.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-04-20T15:51:49.111 INFO:teuthology.orchestra.run.smithi071.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.1 MB/s | 677 kB 00:00 2024-04-20T15:51:49.112 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/25 2024-04-20T15:51:49.152 INFO:teuthology.orchestra.run.smithi071.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 658 kB/s | 44 kB 00:00 2024-04-20T15:51:49.311 INFO:teuthology.orchestra.run.smithi071.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 235 kB/s | 47 kB 00:00 2024-04-20T15:51:49.367 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 4/25 2024-04-20T15:51:49.587 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/25 2024-04-20T15:51:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 95 kB/s | 59 kB 00:00 2024-04-20T15:51:49.986 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/25 2024-04-20T15:51:50.147 INFO:teuthology.orchestra.run.smithi071.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 343 kB/s | 75 kB 00:00 2024-04-20T15:51:50.167 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/25 2024-04-20T15:51:50.179 INFO:teuthology.orchestra.run.smithi071.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 298 kB/s | 306 kB 00:01 2024-04-20T15:51:50.383 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/25 2024-04-20T15:51:50.537 INFO:teuthology.orchestra.run.smithi071.stdout:(20/23): librados2-17.2.7-902.g7f4fa80c.el8.x86 9.7 MB/s | 3.5 MB 00:00 2024-04-20T15:51:50.639 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-20T15:51:50.792 INFO:teuthology.orchestra.run.smithi071.stdout:(21/23): librbd1-17.2.7-902.g7f4fa80c.el8.x86_6 14 MB/s | 3.6 MB 00:00 2024-04-20T15:51:50.860 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-20T15:51:51.539 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-20T15:51:51.692 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 10/25 2024-04-20T15:51:51.756 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 10/25 2024-04-20T15:51:51.960 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:17.2.7-902.g7f4fa80c.el8.x 11/25 2024-04-20T15:51:52.096 INFO:teuthology.orchestra.run.smithi071.stdout:(22/23): ceph-radosgw-17.2.7-902.g7f4fa80c.el8. 3.1 MB/s | 17 MB 00:05 2024-04-20T15:51:52.112 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 12/25 2024-04-20T15:51:52.311 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 13/25 2024-04-20T15:51:52.514 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 14/25 2024-04-20T15:51:52.613 INFO:teuthology.orchestra.run.smithi071.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 720 kB/s | 1.7 MB 00:02 2024-04-20T15:51:52.617 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:51:52.617 INFO:teuthology.orchestra.run.smithi071.stdout:Total 9.9 MB/s | 64 MB 00:06 2024-04-20T15:51:52.658 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:51:52.659 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 15/25 2024-04-20T15:51:52.704 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:51:52.705 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:51:52.729 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 15/25 2024-04-20T15:51:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-20T15:51:53.076 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:51:53.079 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:51:53.133 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-20T15:51:53.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-20T15:51:53.551 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86 19/25 2024-04-20T15:51:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:51:54.312 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-20T15:51:54.504 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T15:51:54.885 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T15:51:55.046 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/25 2024-04-20T15:51:55.146 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/25 2024-04-20T15:51:55.412 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 4/25 2024-04-20T15:51:55.571 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/25 2024-04-20T15:51:55.971 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/25 2024-04-20T15:51:56.138 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/25 2024-04-20T15:51:56.191 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 20/25 2024-04-20T15:51:56.334 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/25 2024-04-20T15:51:56.384 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 20/25 2024-04-20T15:51:56.568 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-20T15:51:56.772 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-20T15:51:56.973 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 20/25 2024-04-20T15:51:57.413 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 21/25 2024-04-20T15:51:57.439 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-20T15:51:57.521 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 21/25 2024-04-20T15:51:57.700 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 22/25 2024-04-20T15:51:57.737 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 10/25 2024-04-20T15:51:57.789 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 10/25 2024-04-20T15:51:57.965 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:17.2.7-902.g7f4fa80c.el8.x 11/25 2024-04-20T15:51:58.115 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 12/25 2024-04-20T15:51:58.305 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 13/25 2024-04-20T15:51:58.547 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 14/25 2024-04-20T15:51:58.670 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 15/25 2024-04-20T15:51:58.729 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 15/25 2024-04-20T15:51:58.930 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-20T15:51:59.162 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-20T15:51:59.356 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-20T15:51:59.546 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86 19/25 2024-04-20T15:52:02.246 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 20/25 2024-04-20T15:52:02.479 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 20/25 2024-04-20T15:52:03.079 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 20/25 2024-04-20T15:52:03.363 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 21/25 2024-04-20T15:52:03.468 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 21/25 2024-04-20T15:52:03.661 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 22/25 2024-04-20T15:52:16.961 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 22/25 2024-04-20T15:52:16.961 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /run 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2024-04-20T15:52:16.962 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:18.643 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 23/25 2024-04-20T15:52:18.709 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 23/25 2024-04-20T15:52:18.709 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T15:52:18.915 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T15:52:18.915 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T15:52:19.645 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T15:52:19.645 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 4/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 7/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:17.2.7-902.g7f4fa80c.el8.x 8/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86 9/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 10/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 11/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 12/25 2024-04-20T15:52:19.646 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 13/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 22/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-20T15:52:19.647 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 24/25 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T15:52:20.503 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:20.505 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-20T15:52:20.505 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:20.505 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:52:20.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-04-20T15:52:21.515 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:44 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:52:22.966 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 22/25 2024-04-20T15:52:22.966 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-04-20T15:52:22.966 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-04-20T15:52:22.966 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2024-04-20T15:52:22.966 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /run 2024-04-20T15:52:22.966 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-04-20T15:52:22.967 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-04-20T15:52:22.967 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-04-20T15:52:22.967 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-04-20T15:52:22.967 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /dev 2024-04-20T15:52:22.967 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:23.125 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:52:23.126 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:23.126 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 53 M 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 179 k 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:52:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:23.128 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-04-20T15:52:23.128 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:23.128 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 54 M 2024-04-20T15:52:23.128 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 236 M 2024-04-20T15:52:23.128 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:52:23.432 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 846 kB/s | 203 kB 00:00 2024-04-20T15:52:23.500 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): libcephsqlite-17.2.7-902.g7f4fa80c.el8.x 582 kB/s | 179 kB 00:00 2024-04-20T15:52:23.517 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-20T15:52:23.583 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-04-20T15:52:23.750 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2024-04-20T15:52:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 23/25 2024-04-20T15:52:24.749 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 23/25 2024-04-20T15:52:24.749 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T15:52:24.942 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T15:52:24.942 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T15:52:25.659 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T15:52:25.659 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/25 2024-04-20T15:52:25.659 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/25 2024-04-20T15:52:25.659 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/25 2024-04-20T15:52:25.659 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 4/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 7/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:17.2.7-902.g7f4fa80c.el8.x 8/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86 9/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 10/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 11/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 12/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 13/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-20T15:52:25.660 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-20T15:52:25.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-20T15:52:25.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-20T15:52:25.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-20T15:52:25.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 22/25 2024-04-20T15:52:25.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-20T15:52:25.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 24/25 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.641 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:26.642 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:52:26.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-04-20T15:52:26.967 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): ceph-test-17.2.7-902.g7f4fa80c.el8.x86_6 14 MB/s | 53 MB 00:03 2024-04-20T15:52:26.969 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:52:26.969 INFO:teuthology.orchestra.run.smithi119.stdout:Total 14 MB/s | 54 MB 00:03 2024-04-20T15:52:27.012 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:52:27.032 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:52:27.032 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:52:27.368 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:52:27.372 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:52:27.666 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:45 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:52:27.856 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:52:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-20T15:52:28.255 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-20T15:52:28.394 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-20T15:52:28.488 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-20T15:52:28.660 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-20T15:52:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/6 2024-04-20T15:52:29.309 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:52:29.310 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:52:29.310 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:52:29.310 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 53 M 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 179 k 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:52:29.311 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:52:29.312 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-04-20T15:52:29.312 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:29.312 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 54 M 2024-04-20T15:52:29.312 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 236 M 2024-04-20T15:52:29.312 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:52:29.590 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 933 kB/s | 203 kB 00:00 2024-04-20T15:52:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-04-20T15:52:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): libcephsqlite-17.2.7-902.g7f4fa80c.el8.x 522 kB/s | 179 kB 00:00 2024-04-20T15:52:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-04-20T15:52:29.882 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 423 kB/s | 71 kB 00:00 2024-04-20T15:52:32.665 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-17.2.7-902.g7f4fa80c.el8.x86_6 16 MB/s | 53 MB 00:03 2024-04-20T15:52:32.667 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:52:32.668 INFO:teuthology.orchestra.run.smithi071.stdout:Total 16 MB/s | 54 MB 00:03 2024-04-20T15:52:32.708 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:52:32.729 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:52:32.729 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:52:33.062 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:52:33.064 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:52:33.535 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:52:33.751 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-20T15:52:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-20T15:52:33.932 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/6 2024-04-20T15:52:34.063 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-20T15:52:34.136 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-20T15:52:34.158 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/6 2024-04-20T15:52:34.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-20T15:52:34.439 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/6 2024-04-20T15:52:34.440 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/6 2024-04-20T15:52:34.440 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/6 2024-04-20T15:52:34.440 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-20T15:52:34.440 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-20T15:52:34.440 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-20T15:52:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/6 2024-04-20T15:52:34.904 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-20T15:52:34.904 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-8.el8.x86_64 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:34.905 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:52:35.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-04-20T15:52:35.855 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:58 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:52:37.498 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:52:37.501 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:37.501 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:52:37.501 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:37.501 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 6.4 k 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 2.3 M 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 1.6 M 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 240 k 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 4.0 M 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 17 M 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-20T15:52:37.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T15:52:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:37.504 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:52:37.505 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:37.505 INFO:teuthology.orchestra.run.smithi119.stdout:Install 32 Packages 2024-04-20T15:52:37.505 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:37.505 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 29 M 2024-04-20T15:52:37.505 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 113 M 2024-04-20T15:52:37.506 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:52:38.026 INFO:teuthology.orchestra.run.smithi119.stdout:(1/32): ceph-17.2.7-902.g7f4fa80c.el8.x86_64.rp 49 kB/s | 6.4 kB 00:00 2024-04-20T15:52:38.801 INFO:teuthology.orchestra.run.smithi119.stdout:(2/32): ceph-mgr-17.2.7-902.g7f4fa80c.el8.x86_6 1.7 MB/s | 1.6 MB 00:00 2024-04-20T15:52:38.910 INFO:teuthology.orchestra.run.smithi119.stdout:(3/32): ceph-mon-17.2.7-902.g7f4fa80c.el8.x86_6 4.6 MB/s | 4.0 MB 00:00 2024-04-20T15:52:39.010 INFO:teuthology.orchestra.run.smithi119.stdout:(4/32): ceph-mgr-modules-core-17.2.7-902.g7f4fa 2.3 MB/s | 240 kB 00:00 2024-04-20T15:52:39.093 INFO:teuthology.orchestra.run.smithi119.stdout:(5/32): ceph-mds-17.2.7-902.g7f4fa80c.el8.x86_6 1.9 MB/s | 2.3 MB 00:01 2024-04-20T15:52:39.202 INFO:teuthology.orchestra.run.smithi119.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 817 kB/s | 157 kB 00:00 2024-04-20T15:52:39.227 INFO:teuthology.orchestra.run.smithi119.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 290 kB/s | 39 kB 00:00 2024-04-20T15:52:39.269 INFO:teuthology.orchestra.run.smithi119.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-04-20T15:52:39.311 INFO:teuthology.orchestra.run.smithi119.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 682 kB/s | 57 kB 00:00 2024-04-20T15:52:39.387 INFO:teuthology.orchestra.run.smithi119.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-04-20T15:52:39.565 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/6 2024-04-20T15:52:39.810 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/6 2024-04-20T15:52:39.895 INFO:teuthology.orchestra.run.smithi119.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 426 kB/s | 249 kB 00:00 2024-04-20T15:52:40.004 INFO:teuthology.orchestra.run.smithi119.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-04-20T15:52:40.046 INFO:teuthology.orchestra.run.smithi119.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 973 kB/s | 641 kB 00:00 2024-04-20T15:52:40.087 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 6/6 2024-04-20T15:52:40.087 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/6 2024-04-20T15:52:40.087 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/6 2024-04-20T15:52:40.087 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-20T15:52:40.087 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-20T15:52:40.087 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-20T15:52:40.163 INFO:teuthology.orchestra.run.smithi119.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2024-04-20T15:52:40.205 INFO:teuthology.orchestra.run.smithi119.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.2 MB/s | 185 kB 00:00 2024-04-20T15:52:40.255 INFO:teuthology.orchestra.run.smithi119.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.8 MB/s | 173 kB 00:00 2024-04-20T15:52:40.289 INFO:teuthology.orchestra.run.smithi119.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 335 kB/s | 11 kB 00:00 2024-04-20T15:52:40.331 INFO:teuthology.orchestra.run.smithi119.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 418 kB/s | 18 kB 00:00 2024-04-20T15:52:40.356 INFO:teuthology.orchestra.run.smithi119.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-04-20T15:52:40.373 INFO:teuthology.orchestra.run.smithi119.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2024-04-20T15:52:40.390 INFO:teuthology.orchestra.run.smithi119.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2024-04-20T15:52:40.424 INFO:teuthology.orchestra.run.smithi119.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 468 kB/s | 16 kB 00:00 2024-04-20T15:52:40.449 INFO:teuthology.orchestra.run.smithi119.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2024-04-20T15:52:40.466 INFO:teuthology.orchestra.run.smithi119.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 447 kB/s | 19 kB 00:00 2024-04-20T15:52:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 722 kB/s | 24 kB 00:00 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-8.el8.x86_64 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:40.497 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:52:40.500 INFO:teuthology.orchestra.run.smithi119.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 871 kB/s | 29 kB 00:00 2024-04-20T15:52:40.517 INFO:teuthology.orchestra.run.smithi119.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 818 kB/s | 27 kB 00:00 2024-04-20T15:52:40.550 INFO:teuthology.orchestra.run.smithi119.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 5.0 MB/s | 255 kB 00:00 2024-04-20T15:52:40.575 INFO:teuthology.orchestra.run.smithi119.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2024-04-20T15:52:40.592 INFO:teuthology.orchestra.run.smithi119.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2024-04-20T15:52:40.609 INFO:teuthology.orchestra.run.smithi119.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 685 kB/s | 23 kB 00:00 2024-04-20T15:52:40.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-04-20T15:52:41.487 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:59 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:52:41.643 INFO:teuthology.orchestra.run.smithi119.stdout:(32/32): ceph-osd-17.2.7-902.g7f4fa80c.el8.x86_ 6.0 MB/s | 17 MB 00:02 2024-04-20T15:52:41.647 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:52:41.647 INFO:teuthology.orchestra.run.smithi119.stdout:Total 7.0 MB/s | 29 MB 00:04 2024-04-20T15:52:41.727 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:52:41.765 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:52:41.765 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:52:42.065 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:52:42.068 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:52:43.136 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:52:43.139 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 6.4 k 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 2.3 M 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 1.6 M 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 240 k 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 4.0 M 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 17 M 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T15:52:43.140 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T15:52:43.141 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T15:52:43.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:Install 32 Packages 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 29 M 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 113 M 2024-04-20T15:52:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:52:43.166 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:52:43.365 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-20T15:52:43.517 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-17.2.7-902.g7f4fa80c.el8.x86_64.rp 34 kB/s | 6.4 kB 00:00 2024-04-20T15:52:43.616 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-20T15:52:43.817 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-20T15:52:44.061 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-20T15:52:44.279 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-20T15:52:44.367 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-mgr-17.2.7-902.g7f4fa80c.el8.x86_6 1.5 MB/s | 1.6 MB 00:01 2024-04-20T15:52:44.452 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-20T15:52:44.628 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-20T15:52:44.709 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): ceph-mds-17.2.7-902.g7f4fa80c.el8.x86_6 1.7 MB/s | 2.3 MB 00:01 2024-04-20T15:52:44.789 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-20T15:52:44.868 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): ceph-mgr-modules-core-17.2.7-902.g7f4fa 1.5 MB/s | 240 kB 00:00 2024-04-20T15:52:45.040 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-20T15:52:45.085 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 725 kB/s | 157 kB 00:00 2024-04-20T15:52:45.143 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 665 kB/s | 39 kB 00:00 2024-04-20T15:52:45.210 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-04-20T15:52:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-20T15:52:45.294 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): ceph-mon-17.2.7-902.g7f4fa80c.el8.x86_6 2.3 MB/s | 4.0 MB 00:01 2024-04-20T15:52:45.320 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 524 kB/s | 57 kB 00:00 2024-04-20T15:52:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-20T15:52:45.536 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.8 MB/s | 457 kB 00:00 2024-04-20T15:52:45.689 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-20T15:52:45.912 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 420 kB/s | 249 kB 00:00 2024-04-20T15:52:46.020 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-04-20T15:52:46.095 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-20T15:52:46.246 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 905 kB/s | 641 kB 00:00 2024-04-20T15:52:46.344 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-20T15:52:46.354 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 131 kB/s | 44 kB 00:00 2024-04-20T15:52:46.529 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-20T15:52:46.621 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 648 kB/s | 173 kB 00:00 2024-04-20T15:52:46.738 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 376 kB/s | 185 kB 00:00 2024-04-20T15:52:46.773 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-20T15:52:47.009 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-20T15:52:47.022 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): ceph-osd-17.2.7-902.g7f4fa80c.el8.x86_ 6.5 MB/s | 17 MB 00:02 2024-04-20T15:52:47.047 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 36 kB/s | 11 kB 00:00 2024-04-20T15:52:47.072 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 853 kB/s | 384 kB 00:00 2024-04-20T15:52:47.156 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 132 kB/s | 18 kB 00:00 2024-04-20T15:52:47.206 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 439 kB/s | 59 kB 00:00 2024-04-20T15:52:47.291 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-20T15:52:47.331 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 125 kB/s | 16 kB 00:00 2024-04-20T15:52:47.407 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-04-20T15:52:47.471 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-20T15:52:47.473 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 132 kB/s | 19 kB 00:00 2024-04-20T15:52:47.507 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): python3-logutils-0.3.5-11.el8.noarch.r 106 kB/s | 49 kB 00:00 2024-04-20T15:52:47.532 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 190 kB/s | 24 kB 00:00 2024-04-20T15:52:47.591 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 249 kB/s | 29 kB 00:00 2024-04-20T15:52:47.660 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-20T15:52:47.686 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 154 kB/s | 27 kB 00:00 2024-04-20T15:52:47.717 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2024-04-20T15:52:47.742 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2024-04-20T15:52:47.826 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 212 kB/s | 23 kB 00:00 2024-04-20T15:52:47.846 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-20T15:52:47.917 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 369 kB/s | 85 kB 00:00 2024-04-20T15:52:47.919 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:52:47.919 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.1 MB/s | 29 MB 00:04 2024-04-20T15:52:48.002 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:52:48.039 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:52:48.039 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:52:48.113 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-20T15:52:48.344 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-20T15:52:48.354 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:52:48.357 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:52:48.558 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-20T15:52:48.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-20T15:52:49.203 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-20T15:52:49.461 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:52:49.551 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-902.g7f4fa80c.el8.n 27/32 2024-04-20T15:52:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-20T15:52:49.729 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 28/32 2024-04-20T15:52:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-20T15:52:50.134 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-20T15:52:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-20T15:52:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-20T15:52:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-20T15:52:50.987 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-20T15:52:51.140 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-20T15:52:51.237 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 28/32 2024-04-20T15:52:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-20T15:52:51.421 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 29/32 2024-04-20T15:52:51.586 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-20T15:52:51.815 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-20T15:52:51.879 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 29/32 2024-04-20T15:52:52.039 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-20T15:52:52.429 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-20T15:52:52.678 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-20T15:52:52.741 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 30/32 2024-04-20T15:52:52.937 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-20T15:52:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 30/32 2024-04-20T15:52:53.129 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 31/32 2024-04-20T15:52:53.146 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 31/32 2024-04-20T15:52:53.150 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-20T15:52:53.290 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 32/32 2024-04-20T15:52:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 32/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 4/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-902.g7f4fa80c.el8.n 6/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-20T15:52:53.686 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-20T15:52:53.687 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-20T15:52:53.688 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-20T15:52:53.705 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-20T15:52:53.898 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-20T15:52:54.099 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-20T15:52:54.272 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-20T15:52:54.487 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-20T15:52:54.487 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:54.487 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:52:54.487 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T15:52:54.488 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T15:52:54.489 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T15:52:54.490 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T15:52:54.490 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T15:52:54.490 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T15:52:54.490 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:54.490 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:52:54.493 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-20T15:52:54.763 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-20T15:52:54.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-04-20T15:52:54.955 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-20T15:52:55.252 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-20T15:52:55.495 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:52:55.572 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-20T15:52:55.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-902.g7f4fa80c.el8.n 27/32 2024-04-20T15:52:56.159 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 28/32 2024-04-20T15:52:57.079 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:52:57.128 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:52:57.130 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:52:57.130 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:52:57.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-04-20T15:52:57.673 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 28/32 2024-04-20T15:52:57.842 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 29/32 2024-04-20T15:52:57.957 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:20 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:52:58.292 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 29/32 2024-04-20T15:52:58.708 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 30/32 2024-04-20T15:52:58.926 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 30/32 2024-04-20T15:52:59.086 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 31/32 2024-04-20T15:52:59.103 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 31/32 2024-04-20T15:52:59.207 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 32/32 2024-04-20T15:52:59.604 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 82 k 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:52:59.605 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:52:59.606 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 82 k 2024-04-20T15:52:59.606 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 372 k 2024-04-20T15:52:59.606 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 32/32 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/32 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/32 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 3/32 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 4/32 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 5/32 2024-04-20T15:52:59.646 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-902.g7f4fa80c.el8.n 6/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-20T15:52:59.647 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-20T15:52:59.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-20T15:52:59.765 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-17.2.7-902.g7f4fa80c.el8.noarch.rpm 515 kB/s | 82 kB 00:00 2024-04-20T15:52:59.765 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:52:59.765 INFO:teuthology.orchestra.run.smithi119.stdout:Total 512 kB/s | 82 kB 00:00 2024-04-20T15:52:59.765 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:52:59.773 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:52:59.773 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:52:59.775 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:52:59.778 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:52:59.823 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:00.299 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:00.452 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-20T15:53:00.452 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T15:53:00.453 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T15:53:00.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:00.455 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:00.490 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:00.714 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:00.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-04-20T15:53:01.086 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:01.086 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:01.086 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:01.086 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:01.086 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:01.087 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:01.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-04-20T15:53:01.474 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:19 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:02.060 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:25 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:03.051 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:53:03.099 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:03.100 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:53:03.101 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:03.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-04-20T15:53:03.682 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 162 k 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:03.683 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:53:03.684 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:03.684 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 162 k 2024-04-20T15:53:03.684 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 494 k 2024-04-20T15:53:03.684 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:03.956 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:21 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:04.803 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-17.2.7-902.g7f4fa80 145 kB/s | 162 kB 00:01 2024-04-20T15:53:04.803 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:04.804 INFO:teuthology.orchestra.run.smithi119.stdout:Total 144 kB/s | 162 kB 00:01 2024-04-20T15:53:04.804 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:04.816 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:04.817 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:04.868 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:04.871 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:05.065 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:05.231 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.e 1/1 2024-04-20T15:53:05.583 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:05.584 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:05.584 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:53:05.585 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:05.585 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:05.585 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 82 k 2024-04-20T15:53:05.585 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:05.585 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:05.586 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:05.586 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:53:05.586 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:05.586 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 82 k 2024-04-20T15:53:05.586 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 372 k 2024-04-20T15:53:05.587 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:05.646 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.e 1/1 2024-04-20T15:53:06.021 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-17.2.7-902.g7f4fa80c.el8.noarch.rpm 188 kB/s | 82 kB 00:00 2024-04-20T15:53:06.022 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:06.022 INFO:teuthology.orchestra.run.smithi071.stdout:Total 187 kB/s | 82 kB 00:00 2024-04-20T15:53:06.022 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:06.028 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.e 1/1 2024-04-20T15:53:06.029 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:06.029 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:06.029 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:06.029 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:06.029 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:06.029 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:06.030 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:06.032 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:06.035 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:06.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-04-20T15:53:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:06.670 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:06.896 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:06.979 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:29 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:07.270 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:07.271 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:07.271 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:07.271 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:07.271 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:07.271 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:07.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-04-20T15:53:08.233 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:26 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:08.556 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:53:08.604 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:08.605 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:53:08.605 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:08.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T15:53:09.426 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:32 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:09.836 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:09.837 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:09.837 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-04-20T15:53:09.837 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:09.837 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:09.837 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 162 k 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 162 k 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 494 k 2024-04-20T15:53:09.838 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:10.692 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-17.2.7-902.g7f4fa80 189 kB/s | 162 kB 00:00 2024-04-20T15:53:10.692 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:10.693 INFO:teuthology.orchestra.run.smithi071.stdout:Total 189 kB/s | 162 kB 00:00 2024-04-20T15:53:10.693 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:10.705 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:10.705 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:10.758 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:10.761 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:10.874 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:11.022 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.e 1/1 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 3.4 M 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 24 k 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 14 k 2024-04-20T15:53:11.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.7 M 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 80 M 2024-04-20T15:53:11.064 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:11.440 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.e 1/1 2024-04-20T15:53:11.764 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): ceph-prometheus-alerts-17.2.7-902.g7f4fa 91 kB/s | 14 kB 00:00 2024-04-20T15:53:11.794 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.e 1/1 2024-04-20T15:53:11.794 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:11.794 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:11.794 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:11.795 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:11.881 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): ceph-grafana-dashboards-17.2.7-902.g7f4f 88 kB/s | 24 kB 00:00 2024-04-20T15:53:11.915 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 283 kB/s | 43 kB 00:00 2024-04-20T15:53:12.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-04-20T15:53:12.123 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): ceph-mgr-dashboard-17.2.7-902.g7f4fa80c. 6.5 MB/s | 3.4 MB 00:00 2024-04-20T15:53:12.232 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 96 kB/s | 34 kB 00:00 2024-04-20T15:53:12.499 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 337 kB/s | 196 kB 00:00 2024-04-20T15:53:12.503 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:12.503 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.5 MB/s | 3.7 MB 00:01 2024-04-20T15:53:12.515 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:12.524 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:12.525 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:12.618 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:12.621 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:12.760 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:30 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:13.093 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-20T15:53:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-20T15:53:13.498 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-20T15:53:13.618 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-902.g7f4fa80c.el8.no 4/6 2024-04-20T15:53:14.340 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:53:14.389 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:14.390 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:53:14.390 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:14.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T15:53:14.710 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-902.g7f4fa80c.el8.n 5/6 2024-04-20T15:53:14.933 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 6/6 2024-04-20T15:53:15.213 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 6/6 2024-04-20T15:53:15.214 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-902.g7f4fa80c.el8.n 1/6 2024-04-20T15:53:15.214 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 2/6 2024-04-20T15:53:15.214 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-902.g7f4fa80c.el8.no 3/6 2024-04-20T15:53:15.214 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-20T15:53:15.214 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-20T15:53:15.223 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:33 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T15:53:15.709 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T15:53:15.710 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T15:53:15.710 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:15.710 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:15.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T15:53:16.557 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:39 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:16.843 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:16.844 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:16.844 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:53:16.844 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 3.4 M 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 24 k 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 14 k 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-04-20T15:53:16.845 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:16.846 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.7 M 2024-04-20T15:53:16.846 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 80 M 2024-04-20T15:53:16.846 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:17.235 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): ceph-prometheus-alerts-17.2.7-902.g7f4fa 95 kB/s | 14 kB 00:00 2024-04-20T15:53:17.261 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-grafana-dashboards-17.2.7-902.g7f4f 137 kB/s | 24 kB 00:00 2024-04-20T15:53:17.378 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2024-04-20T15:53:17.519 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 130 kB/s | 34 kB 00:00 2024-04-20T15:53:17.620 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 812 kB/s | 196 kB 00:00 2024-04-20T15:53:18.214 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 7.4 M 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-20T15:53:18.215 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Install 8 Packages 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 36 M 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 210 M 2024-04-20T15:53:18.216 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:18.578 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-mgr-dashboard-17.2.7-902.g7f4fa80c. 2.2 MB/s | 3.4 MB 00:01 2024-04-20T15:53:18.580 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:18.580 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.1 MB/s | 3.7 MB 00:01 2024-04-20T15:53:18.591 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:18.600 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:18.601 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:18.699 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:18.703 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:18.897 INFO:teuthology.orchestra.run.smithi119.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.7-902 14 MB/s | 7.4 MB 00:00 2024-04-20T15:53:18.968 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:18.998 INFO:teuthology.orchestra.run.smithi119.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.7 MB/s | 4.9 MB 00:00 2024-04-20T15:53:19.065 INFO:teuthology.orchestra.run.smithi119.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 6.8 MB/s | 4.7 MB 00:00 2024-04-20T15:53:19.090 INFO:teuthology.orchestra.run.smithi119.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-20T15:53:19.184 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-20T15:53:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-20T15:53:19.556 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-20T15:53:19.599 INFO:teuthology.orchestra.run.smithi119.stdout:(5/8): python3-scipy-1.0.0-21.module_el8+762+77 27 MB/s | 14 MB 00:00 2024-04-20T15:53:19.632 INFO:teuthology.orchestra.run.smithi119.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-04-20T15:53:19.691 INFO:teuthology.orchestra.run.smithi119.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-04-20T15:53:19.709 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-902.g7f4fa80c.el8.no 4/6 2024-04-20T15:53:19.991 INFO:teuthology.orchestra.run.smithi119.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 440 kB/s | 172 kB 00:00 2024-04-20T15:53:19.992 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:19.992 INFO:teuthology.orchestra.run.smithi119.stdout:Total 20 MB/s | 36 MB 00:01 2024-04-20T15:53:20.222 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:20.240 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:20.240 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:20.755 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-902.g7f4fa80c.el8.n 5/6 2024-04-20T15:53:20.995 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 6/6 2024-04-20T15:53:21.018 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:21.022 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:21.260 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 6/6 2024-04-20T15:53:21.260 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-902.g7f4fa80c.el8.n 1/6 2024-04-20T15:53:21.260 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 2/6 2024-04-20T15:53:21.260 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-902.g7f4fa80c.el8.no 3/6 2024-04-20T15:53:21.260 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-20T15:53:21.261 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-20T15:53:21.633 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-20T15:53:21.633 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:21.633 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:21.633 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:21.633 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:21.633 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:21.634 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T15:53:21.634 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T15:53:21.634 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T15:53:21.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:21.634 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:21.702 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:21.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T15:53:21.835 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T15:53:21.951 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T15:53:22.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T15:53:22.493 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:40 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:22.690 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T15:53:22.860 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T15:53:23.468 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T15:53:23.610 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T15:53:24.131 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:24.132 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-04-20T15:53:24.132 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:53:24.132 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-04-20T15:53:24.132 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:24.132 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 7.4 M 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout:Install 8 Packages 2024-04-20T15:53:24.133 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:24.134 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 36 M 2024-04-20T15:53:24.134 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 210 M 2024-04-20T15:53:24.134 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:24.276 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T15:53:24.596 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T15:53:24.802 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T15:53:25.142 INFO:teuthology.orchestra.run.smithi071.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 7.1 MB/s | 4.7 MB 00:00 2024-04-20T15:53:25.209 INFO:teuthology.orchestra.run.smithi071.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.7 MB/s | 4.9 MB 00:00 2024-04-20T15:53:25.285 INFO:teuthology.orchestra.run.smithi071.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2024-04-20T15:53:25.343 INFO:teuthology.orchestra.run.smithi071.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2024-04-20T15:53:25.846 INFO:teuthology.orchestra.run.smithi071.stdout:(5/8): python3-scipy-1.0.0-21.module_el8+762+77 25 MB/s | 14 MB 00:00 2024-04-20T15:53:26.221 INFO:teuthology.orchestra.run.smithi071.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 735 kB/s | 645 kB 00:00 2024-04-20T15:53:26.313 INFO:teuthology.orchestra.run.smithi071.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 370 kB/s | 172 kB 00:00 2024-04-20T15:53:26.561 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T15:53:27.338 INFO:teuthology.orchestra.run.smithi071.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.7-902 2.6 MB/s | 7.4 MB 00:02 2024-04-20T15:53:27.339 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:27.339 INFO:teuthology.orchestra.run.smithi071.stdout:Total 11 MB/s | 36 MB 00:03 2024-04-20T15:53:27.573 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:27.591 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:27.591 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:27.776 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-20T15:53:27.967 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:27.970 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:27.984 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c 8/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c 8/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c 1/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T15:53:28.294 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-20T15:53:28.295 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-20T15:53:28.541 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:28.709 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T15:53:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-20T15:53:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:28.725 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-20T15:53:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:28.726 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:28.838 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T15:53:28.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-04-20T15:53:28.959 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T15:53:29.561 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T15:53:29.576 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:52 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:29.726 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T15:53:30.340 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T15:53:30.492 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T15:53:31.203 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T15:53:31.207 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:31.208 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 47 k 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T15:53:31.209 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-04-20T15:53:31.210 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:31.211 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 1.9 M 2024-04-20T15:53:31.211 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 25 M 2024-04-20T15:53:31.211 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:31.674 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 115 kB/s | 20 kB 00:00 2024-04-20T15:53:31.699 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 134 kB/s | 27 kB 00:00 2024-04-20T15:53:31.728 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T15:53:31.849 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 722 kB/s | 126 kB 00:00 2024-04-20T15:53:31.875 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 627 kB/s | 110 kB 00:00 2024-04-20T15:53:31.900 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): ceph-mgr-rook-17.2.7-902.g7f4fa80c.el8. 118 kB/s | 47 kB 00:00 2024-04-20T15:53:31.950 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T15:53:32.259 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 92 kB/s | 33 kB 00:00 2024-04-20T15:53:32.284 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 104 kB/s | 43 kB 00:00 2024-04-20T15:53:32.376 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2024-04-20T15:53:32.509 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 235 kB/s | 155 kB 00:00 2024-04-20T15:53:32.785 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 162 kB/s | 81 kB 00:00 2024-04-20T15:53:32.860 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2024-04-20T15:53:32.927 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 154 kB/s | 64 kB 00:00 2024-04-20T15:53:32.952 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 367 kB/s | 61 kB 00:00 2024-04-20T15:53:32.955 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:32.956 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-04-20T15:53:33.012 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:33.027 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:33.027 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:33.115 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:33.118 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:33.699 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T15:53:33.708 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:33.953 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-20T15:53:34.133 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-20T15:53:34.395 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-20T15:53:34.632 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-20T15:53:34.832 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-20T15:53:34.928 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-20T15:53:35.038 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-20T15:53:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c 8/8 2024-04-20T15:53:35.322 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-20T15:53:35.437 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c 8/8 2024-04-20T15:53:35.437 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c 1/8 2024-04-20T15:53:35.438 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-20T15:53:35.438 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-20T15:53:35.438 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T15:53:35.438 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T15:53:35.438 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-20T15:53:35.438 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-20T15:53:35.589 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-20T15:53:35.865 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-20T15:53:35.865 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:35.865 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:35.865 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:35.866 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:36.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-04-20T15:53:36.630 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-20T15:53:36.726 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:36.898 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-20T15:53:37.090 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-20T15:53:37.300 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-20T15:53:37.475 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 13/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 13/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 1/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-20T15:53:37.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-20T15:53:37.772 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-20T15:53:37.772 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-20T15:53:37.772 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-20T15:53:37.772 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-20T15:53:37.772 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-20T15:53:38.216 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-20T15:53:38.216 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:38.216 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:38.217 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:38.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-04-20T15:53:38.352 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:38.354 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:38.354 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:53:38.354 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:38.354 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 47 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T15:53:38.355 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T15:53:38.356 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:38.356 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:38.356 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:38.356 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-04-20T15:53:38.356 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:38.356 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 1.9 M 2024-04-20T15:53:38.357 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 25 M 2024-04-20T15:53:38.357 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:39.057 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:02 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:39.068 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): ceph-mgr-rook-17.2.7-902.g7f4fa80c.el8. 285 kB/s | 47 kB 00:00 2024-04-20T15:53:39.093 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 104 kB/s | 20 kB 00:00 2024-04-20T15:53:39.118 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 123 kB/s | 27 kB 00:00 2024-04-20T15:53:39.311 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 507 kB/s | 110 kB 00:00 2024-04-20T15:53:39.361 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 639 kB/s | 155 kB 00:00 2024-04-20T15:53:39.394 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 387 kB/s | 126 kB 00:00 2024-04-20T15:53:39.495 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 247 kB/s | 33 kB 00:00 2024-04-20T15:53:39.520 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 135 kB/s | 16 kB 00:00 2024-04-20T15:53:39.545 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 182 kB/s | 43 kB 00:00 2024-04-20T15:53:39.599 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 781 kB/s | 81 kB 00:00 2024-04-20T15:53:39.654 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2024-04-20T15:53:39.713 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 6.1 MB/s | 1.2 MB 00:00 2024-04-20T15:53:39.738 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 334 kB/s | 64 kB 00:00 2024-04-20T15:53:39.740 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:39.741 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-04-20T15:53:39.798 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:39.813 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:39.814 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:39.902 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:39.905 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:40.503 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:40.686 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 123 k 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-20T15:53:40.687 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout:Install 5 Packages 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:40.688 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 6.0 M 2024-04-20T15:53:40.689 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-04-20T15:53:40.689 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:40.741 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-20T15:53:40.947 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-20T15:53:41.166 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-20T15:53:41.388 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-20T15:53:41.635 INFO:teuthology.orchestra.run.smithi119.stdout:(1/5): ceph-mgr-cephadm-17.2.7-902.g7f4fa80c.el 227 kB/s | 123 kB 00:00 2024-04-20T15:53:41.645 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-20T15:53:41.660 INFO:teuthology.orchestra.run.smithi119.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.0 MB/s | 578 kB 00:00 2024-04-20T15:53:41.861 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-20T15:53:41.928 INFO:teuthology.orchestra.run.smithi119.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.7 MB/s | 4.8 MB 00:00 2024-04-20T15:53:41.961 INFO:teuthology.orchestra.run.smithi119.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 201 kB/s | 60 kB 00:00 2024-04-20T15:53:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-20T15:53:42.170 INFO:teuthology.orchestra.run.smithi119.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 891 kB/s | 476 kB 00:00 2024-04-20T15:53:42.171 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:42.171 INFO:teuthology.orchestra.run.smithi119.stdout:Total 4.0 MB/s | 6.0 MB 00:01 2024-04-20T15:53:42.225 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:42.236 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:42.236 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:42.341 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:42.344 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:42.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-20T15:53:42.597 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:43.003 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-20T15:53:43.375 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-20T15:53:43.621 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-20T15:53:43.829 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-20T15:53:43.934 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-20T15:53:44.032 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-20T15:53:44.222 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 13/13 2024-04-20T15:53:44.277 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 13/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 1/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-20T15:53:44.500 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-20T15:53:44.501 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-20T15:53:44.501 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-20T15:53:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-20T15:53:44.809 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 5/5 2024-04-20T15:53:44.965 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T15:53:44.966 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T15:53:44.967 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T15:53:44.967 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T15:53:44.967 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T15:53:44.967 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:44.967 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:45.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-04-20T15:53:45.092 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 5/5 2024-04-20T15:53:45.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/5 2024-04-20T15:53:45.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-20T15:53:45.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-20T15:53:45.093 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-20T15:53:45.472 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:45.473 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:45.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-04-20T15:53:45.800 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:03 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:46.318 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:09 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:47.403 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 123 k 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-20T15:53:47.404 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout:Install 5 Packages 2024-04-20T15:53:47.405 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:47.406 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.0 M 2024-04-20T15:53:47.406 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-04-20T15:53:47.406 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:47.963 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:47.963 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:47.963 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 876 k 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-04-20T15:53:47.964 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:47.965 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 959 k 2024-04-20T15:53:47.965 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 3.1 M 2024-04-20T15:53:47.965 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:48.086 INFO:teuthology.orchestra.run.smithi071.stdout:(1/5): ceph-mgr-cephadm-17.2.7-902.g7f4fa80c.el 468 kB/s | 123 kB 00:00 2024-04-20T15:53:48.187 INFO:teuthology.orchestra.run.smithi071.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.6 MB/s | 578 kB 00:00 2024-04-20T15:53:48.504 INFO:teuthology.orchestra.run.smithi071.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 7.0 MB/s | 4.8 MB 00:00 2024-04-20T15:53:48.552 INFO:teuthology.orchestra.run.smithi071.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 165 kB/s | 60 kB 00:00 2024-04-20T15:53:48.587 INFO:teuthology.orchestra.run.smithi071.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 950 kB/s | 476 kB 00:00 2024-04-20T15:53:48.588 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:48.588 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.1 MB/s | 6.0 MB 00:01 2024-04-20T15:53:48.595 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 189 kB/s | 83 kB 00:00 2024-04-20T15:53:48.639 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:48.650 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:48.650 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:48.755 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:48.758 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:49.031 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:49.145 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-fuse-17.2.7-902.g7f4fa80c.el8.x86_6 884 kB/s | 876 kB 00:00 2024-04-20T15:53:49.146 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:49.147 INFO:teuthology.orchestra.run.smithi119.stdout:Total 812 kB/s | 959 kB 00:01 2024-04-20T15:53:49.158 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:49.180 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:49.180 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:49.270 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:49.274 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:49.401 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-20T15:53:49.439 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:49.715 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-20T15:53:49.931 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/2 2024-04-20T15:53:50.315 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/2 2024-04-20T15:53:50.316 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/2 2024-04-20T15:53:50.353 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-20T15:53:50.689 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-20T15:53:50.704 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-20T15:53:50.704 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:50.704 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:50.704 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-20T15:53:50.704 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:50.704 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:50.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-volume 2024-04-20T15:53:50.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-20T15:53:51.202 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 5/5 2024-04-20T15:53:51.480 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 5/5 2024-04-20T15:53:51.480 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 1/5 2024-04-20T15:53:51.480 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-20T15:53:51.480 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-20T15:53:51.481 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-20T15:53:51.644 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:14 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-20T15:53:52.419 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-20T15:53:52.420 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-20T15:53:52.420 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:52.420 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:52.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-04-20T15:53:53.279 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:53.279 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:53.279 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-20T15:53:53.279 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:53.279 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:53.279 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 267 k 2024-04-20T15:53:53.280 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:53.280 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:53.280 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:53.280 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:53:53.280 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:53.280 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 267 k 2024-04-20T15:53:53.281 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 1.3 M 2024-04-20T15:53:53.281 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:53.558 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:11 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:53.760 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-volume-17.2.7-902.g7f4fa80c.el8.noarch.rpm 558 kB/s | 267 kB 00:00 2024-04-20T15:53:53.760 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:53.761 INFO:teuthology.orchestra.run.smithi119.stdout:Total 557 kB/s | 267 kB 00:00 2024-04-20T15:53:53.761 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:53.771 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:53.771 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:53.808 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:53.811 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:53.988 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:54.231 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:54.633 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:55.002 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:53:55.003 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:55.003 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:55.003 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:53:55.003 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:55.003 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:53:55.167 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:53:55.168 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:55.168 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:53:55.168 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:55.168 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:53:55.168 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 876 k 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-04-20T15:53:55.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:55.170 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 959 k 2024-04-20T15:53:55.170 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 3.1 M 2024-04-20T15:53:55.170 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:53:55.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-04-20T15:53:55.454 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 447 kB/s | 83 kB 00:00 2024-04-20T15:53:55.888 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-17.2.7-902.g7f4fa80c.el8.x86_6 1.4 MB/s | 876 kB 00:00 2024-04-20T15:53:55.889 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:55.889 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.3 MB/s | 959 kB 00:00 2024-04-20T15:53:55.898 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:53:55.919 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:53:55.919 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:53:55.949 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:18 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:53:56.005 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:53:56.008 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:53:56.179 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:53:56.462 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-20T15:53:56.645 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/2 2024-04-20T15:53:57.031 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2/2 2024-04-20T15:53:57.031 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/2 2024-04-20T15:53:57.443 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-20T15:53:57.443 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:57.443 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:53:57.443 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-20T15:53:57.444 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:53:57.444 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:53:57.580 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:53:57.580 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:57.580 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-20T15:53:57.580 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:57.580 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 130 k 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:57.581 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 130 k 2024-04-20T15:53:57.582 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 464 k 2024-04-20T15:53:57.582 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:53:57.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-04-20T15:53:57.763 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-17.2.7-902.g7f4fa80c.el8.x86_64. 715 kB/s | 130 kB 00:00 2024-04-20T15:53:57.764 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:53:57.764 INFO:teuthology.orchestra.run.smithi119.stdout:Total 710 kB/s | 130 kB 00:00 2024-04-20T15:53:57.764 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:53:57.769 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:53:57.769 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:53:57.819 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:53:57.822 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:53:57.941 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:53:58.170 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:53:58.421 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:16 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:53:58.454 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:53:59.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:53:59.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:59.388 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:53:59.388 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:53:59.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:53:59.388 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:00.064 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:00.064 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:00.064 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:54:00.064 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:17.2.7-902.g7f4fa80c.el8 ceph-noarch 267 k 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:54:00.065 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:00.066 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 267 k 2024-04-20T15:54:00.066 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.3 M 2024-04-20T15:54:00.066 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:54:00.267 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-volume-17.2.7-902.g7f4fa80c.el8.noarch.rpm 1.3 MB/s | 267 kB 00:00 2024-04-20T15:54:00.267 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:00.267 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.3 MB/s | 267 kB 00:00 2024-04-20T15:54:00.268 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:54:00.278 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:54:00.278 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:54:00.315 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:54:00.318 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:54:00.502 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:54:00.744 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:54:00.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-04-20T15:54:01.161 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:54:01.530 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:24 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:01.551 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 1/1 2024-04-20T15:54:01.551 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:01.551 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:54:01.551 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:17.2.7-902.g7f4fa80c.el8.noarch 2024-04-20T15:54:01.551 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:01.551 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:01.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-04-20T15:54:02.504 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:20 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:03.125 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:03.173 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:03.174 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:03.175 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:03.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-04-20T15:54:03.997 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:26 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:04.159 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 130 k 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:04.160 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:54:04.161 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:04.161 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:54:04.161 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:04.161 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 130 k 2024-04-20T15:54:04.161 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 464 k 2024-04-20T15:54:04.161 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:54:04.349 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-17.2.7-902.g7f4fa80c.el8.x86_64. 690 kB/s | 130 kB 00:00 2024-04-20T15:54:04.350 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:04.350 INFO:teuthology.orchestra.run.smithi071.stdout:Total 685 kB/s | 130 kB 00:00 2024-04-20T15:54:04.350 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:54:04.356 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:54:04.356 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:54:04.405 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:54:04.408 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:54:04.514 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:54:04.742 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:05.021 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:05.414 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:05.414 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:05.414 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:54:05.414 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:05.414 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:05.414 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:05.635 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 25 k 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 25 k 2024-04-20T15:54:05.637 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 109 k 2024-04-20T15:54:05.637 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:54:05.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-04-20T15:54:06.147 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-17.2.7-902.g7f4fa80c.el8.x86_64 49 kB/s | 25 kB 00:00 2024-04-20T15:54:06.148 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:06.148 INFO:teuthology.orchestra.run.smithi119.stdout:Total 48 kB/s | 25 kB 00:00 2024-04-20T15:54:06.149 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:54:06.152 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:54:06.153 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:54:06.154 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:54:06.158 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:54:06.208 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:54:06.391 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:24 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:06.419 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:06.673 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:07.016 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:07.016 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:07.016 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:54:07.016 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:07.016 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:07.016 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:07.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-04-20T15:54:07.867 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:30 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:07.963 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:08.010 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:08.012 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:08.012 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:08.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-04-20T15:54:08.835 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:26 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:09.447 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:09.495 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:09.496 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:09.496 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:09.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-04-20T15:54:10.338 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:33 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:10.481 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:10.482 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-04-20T15:54:10.482 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:10.482 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:54:10.482 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 25 k 2024-04-20T15:54:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:10.483 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:54:10.483 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:10.483 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:54:10.483 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:10.483 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 25 k 2024-04-20T15:54:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 109 k 2024-04-20T15:54:10.484 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:54:10.626 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-17.2.7-902.g7f4fa80c.el8.x86_64 173 kB/s | 25 kB 00:00 2024-04-20T15:54:10.629 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:10.629 INFO:teuthology.orchestra.run.smithi071.stdout:Total 171 kB/s | 25 kB 00:00 2024-04-20T15:54:10.629 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:54:10.631 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:54:10.632 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:54:10.633 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:54:10.636 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:54:10.690 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:54:10.884 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:11.147 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:11.490 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:11.490 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:11.490 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:54:11.490 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:11.490 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:11.490 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:11.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-04-20T15:54:11.916 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:12.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-04-20T15:54:12.335 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:30 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:35 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:13.907 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:13.954 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:13.955 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:13.955 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:14.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-04-20T15:54:14.388 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:14.436 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:14.437 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:14.438 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:14.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-04-20T15:54:14.793 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:32 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:15.277 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:38 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:16.391 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:16.441 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:16.442 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:16.442 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:16.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-04-20T15:54:16.874 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:16.923 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:16.924 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:16.924 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:17.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-04-20T15:54:17.295 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:35 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:17.763 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:40 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:18.877 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:18.927 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:18.928 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:18.928 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:19.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-04-20T15:54:19.360 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:19.408 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:19.409 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:19.409 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:19.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-04-20T15:54:19.755 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:37 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:20.247 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:43 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:21.330 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:21.377 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:21.378 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:21.378 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:21.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-04-20T15:54:21.814 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:21.863 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:21.864 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-04-20T15:54:21.864 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:21.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-04-20T15:54:22.196 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:40 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:22.727 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:45 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:23.782 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:23.829 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:23.830 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:23.830 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:23.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-04-20T15:54:24.364 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 90 k 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:24.365 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:54:24.366 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:24.366 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:54:24.366 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:24.366 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 90 k 2024-04-20T15:54:24.366 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 268 k 2024-04-20T15:54:24.366 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:54:24.634 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-17.2.7-902.g7f4fa80c.el8.x86_64.rpm 338 kB/s | 90 kB 00:00 2024-04-20T15:54:24.634 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:24.634 INFO:teuthology.orchestra.run.smithi119.stdout:Total 336 kB/s | 90 kB 00:00 2024-04-20T15:54:24.635 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:54:24.644 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:54:24.644 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:54:24.655 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:42 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:24.692 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:54:24.695 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:54:24.806 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:54:24.974 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:25.259 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:25.625 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:25.625 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:25.626 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:54:25.626 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:25.626 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:25.626 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:25.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-04-20T15:54:26.256 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 is already installed. 2024-04-20T15:54:26.304 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:26.305 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:54:26.305 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:26.404 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-04-20T15:54:26.646 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:49 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:27.146 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:45 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:28.290 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 3.5 M 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:54:28.291 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:28.292 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:54:28.292 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:28.292 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.5 M 2024-04-20T15:54:28.292 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 16 M 2024-04-20T15:54:28.292 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:54:28.790 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:28.790 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:28.790 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 90 k 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:54:28.791 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:28.792 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 90 k 2024-04-20T15:54:28.792 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 268 k 2024-04-20T15:54:28.792 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:54:28.966 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-17.2.7-902.g7f4fa80c.el8.x86_64.rpm 525 kB/s | 90 kB 00:00 2024-04-20T15:54:28.966 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:28.966 INFO:teuthology.orchestra.run.smithi071.stdout:Total 521 kB/s | 90 kB 00:00 2024-04-20T15:54:28.966 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:54:28.974 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:54:28.975 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:54:29.024 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:54:29.028 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:54:29.130 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:54:29.312 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:29.598 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:29.754 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-17.2.7-902.g7f4fa80c.el8.x86_64.rpm 2.4 MB/s | 3.5 MB 00:01 2024-04-20T15:54:29.755 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:29.755 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.4 MB/s | 3.5 MB 00:01 2024-04-20T15:54:29.756 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:54:29.769 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:54:29.769 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:54:29.835 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:54:29.838 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:54:29.986 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:29.986 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:29.986 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:54:29.986 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:29.986 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:29.986 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:30.223 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-04-20T15:54:30.306 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:54:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:30.916 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:30.960 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:48 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:31.306 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:54:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:31.306 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:31.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-04-20T15:54:32.259 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:55 ago on Sat 20 Apr 2024 03:51:37 PM UTC. 2024-04-20T15:54:32.602 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:32.603 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:32.603 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-04-20T15:54:32.603 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 3.5 M 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.5 M 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 16 M 2024-04-20T15:54:32.604 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:54:33.183 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-17.2.7-902.g7f4fa80c.el8.x86_64.rpm 6.0 MB/s | 3.5 MB 00:00 2024-04-20T15:54:33.184 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:33.184 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.0 MB/s | 3.5 MB 00:00 2024-04-20T15:54:33.184 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:54:33.196 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:54:33.196 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:54:33.258 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:54:33.260 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:54:33.707 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:54:33.897 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-20T15:54:33.898 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:33.898 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-04-20T15:54:33.898 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:33.898 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-20T15:54:33.898 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 175 k 2024-04-20T15:54:33.898 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 175 k 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 556 k 2024-04-20T15:54:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-20T15:54:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:34.091 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-17.2.7-902.g7f4fa80c.el8.x86_64.rpm 911 kB/s | 175 kB 00:00 2024-04-20T15:54:34.091 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:34.092 INFO:teuthology.orchestra.run.smithi119.stdout:Total 905 kB/s | 175 kB 00:00 2024-04-20T15:54:34.092 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-20T15:54:34.102 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-20T15:54:34.102 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-20T15:54:34.151 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-20T15:54:34.154 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-20T15:54:34.272 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-20T15:54:34.311 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:34.448 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:34.669 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:34.669 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:34.669 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:54:34.669 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:34.669 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:34.669 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:34.737 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:34.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-04-20T15:54:35.137 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:35.137 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:35.137 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-20T15:54:35.137 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:35.138 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:35.138 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-20T15:54:35.343 DEBUG:teuthology.parallel:result is None 2024-04-20T15:54:35.616 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:02:53 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:54:37.245 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:54:37.246 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:17.2.7-902.g7f4fa80c.el8 ceph 175 k 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 175 k 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 556 k 2024-04-20T15:54:37.247 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:54:37.619 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-17.2.7-902.g7f4fa80c.el8.x86_64.rpm 470 kB/s | 175 kB 00:00 2024-04-20T15:54:37.620 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:54:37.620 INFO:teuthology.orchestra.run.smithi071.stdout:Total 468 kB/s | 175 kB 00:00 2024-04-20T15:54:37.620 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:54:37.631 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:54:37.631 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:54:37.680 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:54:37.683 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:54:37.803 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:54:37.977 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:38.260 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:38.610 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 1/1 2024-04-20T15:54:38.611 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:38.611 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:54:38.611 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:17.2.7-902.g7f4fa80c.el8.x86_64 2024-04-20T15:54:38.611 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:38.611 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:54:38.844 DEBUG:teuthology.parallel:result is None 2024-04-20T15:54:38.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7f4fa80c8c97f53f52b1189d2347b5761f6287e3 2024-04-20T15:54:38.982 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T15:54:39.003 INFO:teuthology.orchestra.run.smithi071.stdout:17.2.7-902.g7f4fa80c.el8 2024-04-20T15:54:39.004 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-902.g7f4fa80c.el8 2024-04-20T15:54:39.004 INFO:teuthology.task.install:The correct ceph version 17.2.7-902.g7f4fa80c is installed. 2024-04-20T15:54:39.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7f4fa80c8c97f53f52b1189d2347b5761f6287e3 2024-04-20T15:54:39.141 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T15:54:39.167 INFO:teuthology.orchestra.run.smithi119.stdout:17.2.7-902.g7f4fa80c.el8 2024-04-20T15:54:39.168 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-902.g7f4fa80c.el8 2024-04-20T15:54:39.168 INFO:teuthology.task.install:The correct ceph version 17.2.7-902.g7f4fa80c is installed. 2024-04-20T15:54:39.169 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-20T15:54:39.170 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:39.170 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T15:54:39.196 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:39.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T15:54:39.239 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-20T15:54:39.239 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:39.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T15:54:39.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T15:54:39.331 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:39.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T15:54:39.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T15:54:39.427 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-20T15:54:39.428 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:39.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T15:54:39.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T15:54:39.520 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:39.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T15:54:39.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T15:54:39.615 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-20T15:54:39.615 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:39.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T15:54:39.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T15:54:39.708 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:39.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T15:54:39.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T15:54:39.802 INFO:teuthology.run_tasks:Running task ceph... 2024-04-20T15:54:39.884 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-20T15:54:39.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-04-20T15:54:39.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2024-04-20T15:54:39.914 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-20T15:54:39.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-20T15:54:39.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-20T15:54:39.981 INFO:tasks.ceph:Creating extra log directories... 2024-04-20T15:54:39.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-20T15:54:40.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-20T15:54:40.058 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-20T15:54:40.058 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-20T15:54:40.058 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647', 'branch': 'wip-yuri4-testing-2024-04-19-0708-quincy', 'description': 'rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7664647', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2024-04-19-0708-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3', 'wait-for-scrub': False}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3'}}, 'rgw': {'frontend': 'beast', 'storage classes': 'LUKEWARM, FROZEN'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2024-04-19-0708-quincy', 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 367, 'sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri4-testing-2024-04-19-0708-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7f4fa80c8c97f53f52b1189d2347b5761f6287e3', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDb2dPrGNonFiPBD42oeSD+ltZCSt7KxYjqQLgxirOxBqzNetAzxLREfOYmzF4MpHrzNQitBAOmQXxMU0UXxK+E=', 'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJr7BmfmWrdB1CO79XiqR7yDTi5evHE7CHFwOfOPXCu5jkFmr+QPbFz8h5Sn5L62sRrU2MLudH+ncq8K5mrgzA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None}}, {'rabbitmq': {'client.0': None}}, {'notification-tests': {'client.0': {'extra_attr': ['amqp_test'], 'rgw_server': 'client.0'}}}, {'kafka': {'client.0': {'kafka_version': '2.6.0'}}}, {'notification-tests': {'client.0': {'extra_attr': ['kafka_test'], 'rgw_server': 'client.0'}}}, {'notification-tests': {'client.0': {'rgw_server': 'client.0'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-20_15:31:09', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-20T15:54:40.059 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-20T15:54:40.086 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-20T15:54:40.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-04-20T15:54:40.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-04-20T15:54:40.171 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:40.172 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T15:54:40.224 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T15:54:40.225 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-04-20T15:54:40.282 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 165385 Links: 1 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-20 15:54:34.068072545 +0000 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-20 15:50:35.941584543 +0000 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-20 15:50:35.941584543 +0000 2024-04-20T15:54:40.283 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-20T15:54:40.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T15:54:40.348 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-20T15:54:40.348 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-20T15:54:40.348 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000248089 s, 2.1 MB/s 2024-04-20T15:54:40.349 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T15:54:40.408 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-04-20T15:54:40.466 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 165463 Links: 1 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-20 15:54:34.068072545 +0000 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-20 15:50:36.313578127 +0000 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-20 15:50:36.313578127 +0000 2024-04-20T15:54:40.467 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-20T15:54:40.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T15:54:40.532 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-20T15:54:40.532 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-20T15:54:40.532 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000228959 s, 2.2 MB/s 2024-04-20T15:54:40.533 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T15:54:40.591 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-04-20T15:54:40.648 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T15:54:40.648 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:40.648 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 170200 Links: 1 2024-04-20T15:54:40.648 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:40.648 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:40.649 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-20 15:54:34.068072545 +0000 2024-04-20T15:54:40.649 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-20 15:50:36.701571435 +0000 2024-04-20T15:54:40.649 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-20 15:50:36.701571435 +0000 2024-04-20T15:54:40.649 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-20T15:54:40.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T15:54:40.713 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-20T15:54:40.713 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-20T15:54:40.713 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000159527 s, 3.2 MB/s 2024-04-20T15:54:40.714 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T15:54:40.772 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-04-20T15:54:40.830 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 6h/6d Inode: 169246 Links: 1 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-20 15:54:34.068072545 +0000 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-20 15:50:37.050565416 +0000 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-20 15:50:37.050565416 +0000 2024-04-20T15:54:40.831 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-20T15:54:40.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T15:54:40.899 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-20T15:54:40.899 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-20T15:54:40.899 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000212378 s, 2.4 MB/s 2024-04-20T15:54:40.900 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T15:54:40.959 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:40.959 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:40.959 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T15:54:40.979 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T15:54:40.979 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 169198 Links: 1 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-20 15:54:30.672694983 +0000 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-20 15:50:45.484624054 +0000 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-20 15:50:45.484624054 +0000 2024-04-20T15:54:41.041 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-04-20T15:54:41.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T15:54:41.108 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-20T15:54:41.109 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-20T15:54:41.109 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000227934 s, 2.2 MB/s 2024-04-20T15:54:41.110 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T15:54:41.169 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 168379 Links: 1 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-20 15:54:30.672694983 +0000 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-20 15:50:45.840615431 +0000 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-20 15:50:45.840615431 +0000 2024-04-20T15:54:41.228 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-04-20T15:54:41.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T15:54:41.293 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-20T15:54:41.294 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-20T15:54:41.294 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000254399 s, 2.0 MB/s 2024-04-20T15:54:41.295 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T15:54:41.353 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-04-20T15:54:41.411 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T15:54:41.411 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:41.411 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 170600 Links: 1 2024-04-20T15:54:41.412 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:41.412 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:41.412 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-20 15:54:30.672694983 +0000 2024-04-20T15:54:41.412 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-20 15:50:46.186607051 +0000 2024-04-20T15:54:41.412 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-20 15:50:46.186607051 +0000 2024-04-20T15:54:41.412 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-04-20T15:54:41.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T15:54:41.476 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-20T15:54:41.476 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-20T15:54:41.476 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.00020422 s, 2.5 MB/s 2024-04-20T15:54:41.477 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T15:54:41.535 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-04-20T15:54:41.592 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 168499 Links: 1 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-20 15:54:30.672694983 +0000 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-20 15:50:46.530598720 +0000 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-20 15:50:46.530598720 +0000 2024-04-20T15:54:41.593 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-04-20T15:54:41.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T15:54:41.657 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-20T15:54:41.657 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-20T15:54:41.657 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000189348 s, 2.7 MB/s 2024-04-20T15:54:41.658 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T15:54:41.717 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:41.717 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-20T15:54:41.717 INFO:tasks.ceph:Generating config... 2024-04-20T15:54:41.718 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-20T15:54:41.718 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-20T15:54:41.718 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-20T15:54:41.718 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-04-20T15:54:41.718 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2024-04-20T15:54:41.718 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-20T15:54:41.719 INFO:tasks.ceph:[client] setuser = ceph 2024-04-20T15:54:41.719 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2024-04-20T15:54:41.719 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2024-04-20T15:54:41.719 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-20T15:54:41.719 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-20T15:54:41.719 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-20T15:54:41.719 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-20T15:54:41.719 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-20T15:54:41.719 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-20T15:54:41.719 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-20T15:54:41.719 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-20T15:54:41.719 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-20T15:54:41.719 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-20T15:54:41.719 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-20T15:54:41.720 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-20T15:54:41.720 INFO:tasks.ceph:Setting up mon.a... 2024-04-20T15:54:41.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-20T15:54:41.774 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2024-04-20T15:54:41.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-20T15:54:41.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-20T15:54:41.889 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.71'), ('mon.c', '[v2:172.21.15.71:3301,v1:172.21.15.71:6790]'), ('mon.b', '172.21.15.119')] 2024-04-20T15:54:41.889 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71,[v2:172.21.15.71:3301,v1:172.21.15.71:6790],172.21.15.119', 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-20T15:54:41.890 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:41.890 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-20T15:54:41.947 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --addv c '[v2:172.21.15.71:3301,v1:172.21.15.71:6790]' --add b 172.21.15.119 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:42.032 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:54:42.032 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:54:42.033 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:42.033 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid daea1877-30c6-40c4-b0a7-4fd11c4a648b 2024-04-20T15:54:42.033 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = octopus 2024-04-20T15:54:42.033 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:fsid daea1877-30c6-40c4-b0a7-4fd11c4a648b 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2024-04-20T15:54:42.034340+0000 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:created 2024-04-20T15:54:42.034340+0000 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 15 (octopus) 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:1: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:2: [v2:172.21.15.71:3301/0,v1:172.21.15.71:6790/0] mon.c 2024-04-20T15:54:42.034 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-20T15:54:42.035 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-20T15:54:42.053 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID daea1877-30c6-40c4-b0a7-4fd11c4a648b... 2024-04-20T15:54:42.054 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-20T15:54:42.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-04-20T15:54:42.137 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-04-20T15:54:42.138 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-04-20T15:54:42.139 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.71,[v2:172.21.15.71:3301,v1:172.21.15.71:6790],172.21.15.119" 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: osd_max_pg_log_entries = 10 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: osd_min_pg_log_entries = 10 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = daea1877-30c6-40c4-b0a7-4fd11c4a648b 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-04-20T15:54:42.140 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-04-20T15:54:42.141 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: bdev async discard = True 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: bdev enable discard = True 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore allocator = bitmap 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-04-20T15:54:42.142 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 1/20 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 1/20 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 4/10 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-20T15:54:42.143 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-04-20T15:54:42.144 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-20T15:54:42.145 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: debug rgw = 20 2024-04-20T15:54:42.146 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt require ssl = False 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms backend = testing 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout: rgw lc debug interval = 10 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout: setgroup = ceph 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout: setuser = ceph 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-04-20T15:54:42.147 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-04-20T15:54:42.148 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-04-20T15:54:42.149 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-04-20T15:54:42.150 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.71,[v2:172.21.15.71:3301,v1:172.21.15.71:6790],172.21.15.119" 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd_max_pg_log_entries = 10 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd_min_pg_log_entries = 10 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = daea1877-30c6-40c4-b0a7-4fd11c4a648b 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-04-20T15:54:42.151 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.152 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: bdev async discard = True 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: bdev enable discard = True 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore allocator = bitmap 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 1/20 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 1/20 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-04-20T15:54:42.153 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 4/10 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: osd shutdown pgref assert = True 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-04-20T15:54:42.154 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-04-20T15:54:42.155 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-04-20T15:54:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: debug rgw = 20 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt require ssl = False 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt s3 kms backend = testing 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: rgw lc debug interval = 10 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: setgroup = ceph 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout: setuser = ceph 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-04-20T15:54:42.157 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-04-20T15:54:42.159 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-20T15:54:42.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-20T15:54:42.252 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-20T15:54:42.252 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:42.252 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-20T15:54:42.270 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:42.270 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-20T15:54:42.328 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T15:54:42.328 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:42.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-20T15:54:42.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-20T15:54:42.403 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:54:42.403 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:42.460 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T15:54:42.460 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:42.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-20T15:54:42.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-20T15:54:42.497 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:54:42.497 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:42.555 INFO:tasks.ceph:Setting up mon nodes... 2024-04-20T15:54:42.555 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-20T15:54:42.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-20T15:54:42.604 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-20T15:54:42.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-20T15:54:42.674 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-20T15:54:42.676 INFO:tasks.ceph:Setting up mds nodes... 2024-04-20T15:54:42.676 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-20T15:54:42.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-20T15:54:42.716 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-20T15:54:42.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-20T15:54:42.768 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-20T15:54:42.780 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-20T15:54:42.780 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-20T15:54:42.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-20T15:54:42.807 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:42.807 INFO:tasks.ceph:role: osd.0 2024-04-20T15:54:42.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T15:54:42.807 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:42.881 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:42.882 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:42.882 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:42.882 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:43.309 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-20T15:54:43.311 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-20T15:54:43.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-20T15:54:43.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-20T15:54:43.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-20T15:54:43.570 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:43.570 INFO:tasks.ceph:role: osd.1 2024-04-20T15:54:43.570 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T15:54:43.570 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:43.644 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:44.064 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-20T15:54:44.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-20T15:54:44.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-20T15:54:44.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-20T15:54:44.181 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-20T15:54:44.248 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:44.248 INFO:tasks.ceph:role: osd.2 2024-04-20T15:54:44.248 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T15:54:44.249 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-20T15:54:44.325 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:44.325 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:44.325 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:44.325 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:44.325 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:44.326 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:44.326 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:44.326 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:44.326 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:44.326 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:44.776 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-20T15:54:44.778 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-20T15:54:44.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-20T15:54:44.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-20T15:54:44.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-20T15:54:44.988 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:44.988 INFO:tasks.ceph:role: osd.3 2024-04-20T15:54:44.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T15:54:44.988 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-20T15:54:45.062 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:45.062 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:45.062 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:45.063 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:45.502 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-20T15:54:45.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-20T15:54:45.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-20T15:54:45.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-20T15:54:45.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:45.785 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:45.785+0000 7f7b215ea540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-20T15:54:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:45.788+0000 7f7b215ea540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-20T15:54:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:45.788+0000 7f7b215ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-20T15:54:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:45.788+0000 7f7b215ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-20T15:54:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:45.788+0000 7f7b215ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-20T15:54:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:45.790+0000 7f7b215ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-20T15:54:46.317 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:46.317+0000 7f7b215ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:47.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:47.086+0000 7f7b215ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:47.848 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:47.848+0000 7f7b215ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:48.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T15:54:48.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:48.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:48.761+0000 7fcb65dc9540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-20T15:54:48.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:48.761+0000 7fcb65dc9540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-20T15:54:48.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:48.761+0000 7fcb65dc9540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-20T15:54:48.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:48.761+0000 7fcb65dc9540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-20T15:54:48.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:48.761+0000 7fcb65dc9540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-20T15:54:48.762 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:48.762+0000 7fcb65dc9540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-20T15:54:49.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:49.291+0000 7fcb65dc9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:50.061 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:50.061+0000 7fcb65dc9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:50.823 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:50.823+0000 7fcb65dc9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:51.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T15:54:51.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:51.735+0000 7fdf2ccc4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-20T15:54:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:51.735+0000 7fdf2ccc4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-20T15:54:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:51.735+0000 7fdf2ccc4540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-20T15:54:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:51.735+0000 7fdf2ccc4540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-20T15:54:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:51.735+0000 7fdf2ccc4540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-20T15:54:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:51.736+0000 7fdf2ccc4540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-20T15:54:52.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:52.261+0000 7fdf2ccc4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:53.027 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:53.026+0000 7fdf2ccc4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:53.790 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:53.789+0000 7fdf2ccc4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:54.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T15:54:54.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:54:54.706 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:54.706+0000 7f8cf6503540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-20T15:54:54.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:54.706+0000 7f8cf6503540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-20T15:54:54.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:54.706+0000 7f8cf6503540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-20T15:54:54.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:54.706+0000 7f8cf6503540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-20T15:54:54.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:54.706+0000 7f8cf6503540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-20T15:54:54.707 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:54.706+0000 7f8cf6503540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-20T15:54:55.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:55.231+0000 7f8cf6503540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:55.995 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:55.995+0000 7f8cf6503540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:56.758 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:54:56.758+0000 7f8cf6503540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:54:57.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T15:54:57.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-20T15:54:57.615 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:57.615 INFO:tasks.ceph:role: osd.4 2024-04-20T15:54:57.615 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T15:54:57.615 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:57.694 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:57.695 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:57.695 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:58.128 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-20T15:54:58.130 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-20T15:54:58.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-20T15:54:58.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-20T15:54:58.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-20T15:54:58.378 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:58.378 INFO:tasks.ceph:role: osd.5 2024-04-20T15:54:58.378 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T15:54:58.378 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:58.456 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:58.457 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:58.884 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-20T15:54:58.886 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-20T15:54:58.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-20T15:54:58.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-20T15:54:59.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-20T15:54:59.090 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:59.090 INFO:tasks.ceph:role: osd.6 2024-04-20T15:54:59.090 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T15:54:59.091 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-20T15:54:59.166 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:59.166 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:59.166 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:59.166 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:59.166 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:59.166 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:59.167 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:59.167 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:59.167 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:59.167 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:54:59.627 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-20T15:54:59.628 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-20T15:54:59.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-20T15:54:59.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-20T15:54:59.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-20T15:54:59.850 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T15:54:59.851 INFO:tasks.ceph:role: osd.7 2024-04-20T15:54:59.851 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T15:54:59.851 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-20T15:54:59.925 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T15:54:59.926 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T15:55:00.373 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-04-20T15:55:00.374 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-04-20T15:55:00.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-20T15:55:00.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-20T15:55:00.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:55:00.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:00.650+0000 7fc43143a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-20T15:55:00.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:00.651+0000 7fc43143a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-20T15:55:00.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:00.651+0000 7fc43143a540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-20T15:55:00.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:00.651+0000 7fc43143a540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-20T15:55:00.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:00.651+0000 7fc43143a540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-20T15:55:00.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:00.658+0000 7fc43143a540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-20T15:55:01.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:01.187+0000 7fc43143a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:01.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:01.952+0000 7fc43143a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:02.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:02.715+0000 7fc43143a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:03.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-20T15:55:03.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:55:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:03.628+0000 7ff187687540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-20T15:55:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:03.628+0000 7ff187687540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-20T15:55:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:03.628+0000 7ff187687540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-20T15:55:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:03.628+0000 7ff187687540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-20T15:55:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:03.628+0000 7ff187687540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-20T15:55:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:03.629+0000 7ff187687540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-20T15:55:04.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:04.154+0000 7ff187687540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:04.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:04.916+0000 7ff187687540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:05.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:05.682+0000 7ff187687540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:06.473 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-20T15:55:06.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:55:06.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:06.592+0000 7fd3aa2a2540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-20T15:55:06.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:06.592+0000 7fd3aa2a2540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-20T15:55:06.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:06.592+0000 7fd3aa2a2540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-20T15:55:06.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:06.592+0000 7fd3aa2a2540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-20T15:55:06.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:06.592+0000 7fd3aa2a2540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-20T15:55:06.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:06.593+0000 7fd3aa2a2540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-20T15:55:07.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:07.120+0000 7fd3aa2a2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:07.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:07.883+0000 7fd3aa2a2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:08.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:08.645+0000 7fd3aa2a2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:09.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-20T15:55:09.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:55:09.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:09.556+0000 7f4f0028f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-20T15:55:09.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:09.556+0000 7f4f0028f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-20T15:55:09.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:09.556+0000 7f4f0028f540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-20T15:55:09.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:09.556+0000 7f4f0028f540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-20T15:55:09.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:09.556+0000 7f4f0028f540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-20T15:55:09.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:09.557+0000 7f4f0028f540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-20T15:55:10.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:10.086+0000 7f4f0028f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:10.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:10.853+0000 7f4f0028f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:11.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-20T15:55:11.616+0000 7f4f0028f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-20T15:55:12.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-20T15:55:12.440 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-20T15:55:12.440 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:12.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-20T15:55:12.475 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:12.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-20T15:55:12.544 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:12.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-20T15:55:12.613 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:12.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-20T15:55:12.681 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:12.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-20T15:55:12.750 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:12.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-20T15:55:12.781 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:12.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-20T15:55:12.852 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:12.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-20T15:55:12.922 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:12.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-20T15:55:12.990 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:12.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-20T15:55:13.058 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:13.058 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-20T15:55:13.077 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:13.077 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-20T15:55:13.116 INFO:tasks.ceph:Adding keys to all mons... 2024-04-20T15:55:13.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-20T15:55:13.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-20T15:55:13.159 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2024-04-20T15:55:13.159 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBC5SNmEH4hJBAAfusHRy3N973ZKSekOxxFUQ== 2024-04-20T15:55:13.159 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBF5SNmfi3sLhAAIz8Fma8b/lvWvw/0f4TaLw== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBI5SNm+hl4LRAATrnXV/cmKiy3WMyirViuNQ== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBL5SNmDcjuKxAAGgESojU+QULbEZrQ7q4mZQ== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBO5SNmhQYuKhAAqjYAGF8IAUdRoHkyvCDs7w== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBC5SNmfyE7KBAAGoTkCE36OFv/xq9DpxdiSQ== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBU5SNmvBXhJhAAgR6ZTK1evZEQIBxR+OTQUg== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBX5SNmveqWJRAA+iBT0bLlYRQdkFGdgT2ruQ== 2024-04-20T15:55:13.160 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBa5SNmBO9dIxAAWEtGvK7vZNh2Y9u4tqdoCA== 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBd5SNmGwg6IRAAi6hgHqVxKDwrKq1gNP+wKw== 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBC5SNme0LGKhAA5hvy7c/fq2ZXBCTvBkAebg== 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-04-20T15:55:13.161 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBC5SNmAnfmLRAAtHZwOKA8MpyTYcv3eHBpyA== 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBC5SNmEH4hJBAAfusHRy3N973ZKSekOxxFUQ== 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBF5SNmfi3sLhAAIz8Fma8b/lvWvw/0f4TaLw== 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBI5SNm+hl4LRAATrnXV/cmKiy3WMyirViuNQ== 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBL5SNmDcjuKxAAGgESojU+QULbEZrQ7q4mZQ== 2024-04-20T15:55:13.182 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBO5SNmhQYuKhAAqjYAGF8IAUdRoHkyvCDs7w== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBC5SNmfyE7KBAAGoTkCE36OFv/xq9DpxdiSQ== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBU5SNmvBXhJhAAgR6ZTK1evZEQIBxR+OTQUg== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBX5SNmveqWJRAA+iBT0bLlYRQdkFGdgT2ruQ== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBa5SNmBO9dIxAAWEtGvK7vZNh2Y9u4tqdoCA== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBd5SNmGwg6IRAAi6hgHqVxKDwrKq1gNP+wKw== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBC5SNme0LGKhAA5hvy7c/fq2ZXBCTvBkAebg== 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2024-04-20T15:55:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBC5SNmAnfmLRAAtHZwOKA8MpyTYcv3eHBpyA== 2024-04-20T15:55:13.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T15:55:13.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T15:55:13.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T15:55:13.480 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T15:55:13.529 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.682 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T15:55:13.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T15:55:13.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T15:55:13.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T15:55:13.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T15:55:13.842 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-20T15:55:13.842 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-20T15:55:13.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-20T15:55:13.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-20T15:55:13.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-20T15:55:14.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-20T15:55:14.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-20T15:55:14.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-20T15:55:14.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-20T15:55:14.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-20T15:55:14.321 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:55:14.323 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-20T15:55:14.379 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-20T15:55:14.379 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-20T15:55:14.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-20T15:55:14.381 INFO:tasks.ceph.mon.a:Started 2024-04-20T15:55:14.381 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-20T15:55:14.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-20T15:55:14.383 INFO:tasks.ceph.mon.c:Started 2024-04-20T15:55:14.383 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-20T15:55:14.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-20T15:55:14.420 INFO:tasks.ceph.mon.b:Started 2024-04-20T15:55:14.421 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-20T15:55:14.421 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-20T15:55:14.421 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-20T15:55:14.423 INFO:tasks.ceph.mgr.y:Started 2024-04-20T15:55:14.424 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-20T15:55:14.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-20T15:55:14.426 INFO:tasks.ceph.mgr.x:Started 2024-04-20T15:55:14.426 INFO:tasks.ceph:Setting crush tunables to default 2024-04-20T15:55:14.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2024-04-20T15:55:14.817 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2024-04-20T15:55:14.828 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-20T15:55:14.828 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:14.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-20T15:55:14.849 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:14.850+0000 7f04da0891c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T15:55:14.855 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:14.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-20T15:55:14.922 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:14.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-20T15:55:15.001 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:55:15.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-20T15:55:15.028 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:15.029+0000 7f04da0891c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T15:55:15.055 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:15.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-20T15:55:15.084 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:15.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-20T15:55:15.150 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:15.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-20T15:55:15.217 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-20T15:55:15.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-20T15:55:15.218 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:15.218+0000 7f5ae7f071c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T15:55:15.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new ea8322c2-2529-4b81-94ec-f742309b2184 0 2024-04-20T15:55:15.372 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:15.373+0000 7f04da0891c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T15:55:15.387 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:15.387+0000 7f5ae7f071c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T15:55:15.518 INFO:teuthology.orchestra.run.smithi119.stdout:0 2024-04-20T15:55:15.530 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new c12e91af-fc5b-47fe-b872-687eb056ef9f 1 2024-04-20T15:55:15.761 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:15.762+0000 7f5ae7f071c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T15:55:15.776 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-04-20T15:55:15.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new e9758b96-7c20-49e5-b1e8-b2fb21b567b9 2 2024-04-20T15:55:15.862 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:15.863+0000 7f04da0891c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T15:55:15.998 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:15.999+0000 7f04da0891c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T15:55:16.004 INFO:teuthology.orchestra.run.smithi119.stdout:2 2024-04-20T15:55:16.016 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 2e9693e5-788f-40d2-a638-4bc6ee66a939 3 2024-04-20T15:55:16.231 INFO:teuthology.orchestra.run.smithi119.stdout:3 2024-04-20T15:55:16.243 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new a7519d13-2502-4a64-9af5-35082e560381 4 2024-04-20T15:55:16.288 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:16.288+0000 7f5ae7f071c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T15:55:16.330 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:16.331+0000 7f04da0891c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T15:55:16.438 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:16.438+0000 7f5ae7f071c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T15:55:16.461 INFO:teuthology.orchestra.run.smithi119.stdout:4 2024-04-20T15:55:16.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 8079627a-12e4-4244-bba2-6608e9a58860 5 2024-04-20T15:55:16.689 INFO:teuthology.orchestra.run.smithi119.stdout:5 2024-04-20T15:55:16.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 34517aca-45eb-442f-9270-7bda8fe5af33 6 2024-04-20T15:55:16.796 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:16.796+0000 7f5ae7f071c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T15:55:16.912 INFO:teuthology.orchestra.run.smithi119.stdout:6 2024-04-20T15:55:16.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new d0f29ef3-eb70-4b21-a0fb-bb969904cca0 7 2024-04-20T15:55:17.038 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.039+0000 7f04da0891c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T15:55:17.113 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.111+0000 7f04da0891c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T15:55:17.136 INFO:teuthology.orchestra.run.smithi119.stdout:7 2024-04-20T15:55:17.148 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-20T15:55:17.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-20T15:55:17.152 INFO:tasks.ceph.osd.0:Started 2024-04-20T15:55:17.152 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-20T15:55:17.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-20T15:55:17.154 INFO:tasks.ceph.osd.1:Started 2024-04-20T15:55:17.155 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-20T15:55:17.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-20T15:55:17.157 INFO:tasks.ceph.osd.2:Started 2024-04-20T15:55:17.157 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-20T15:55:17.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-20T15:55:17.160 INFO:tasks.ceph.osd.3:Started 2024-04-20T15:55:17.160 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-20T15:55:17.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-20T15:55:17.163 INFO:tasks.ceph.osd.4:Started 2024-04-20T15:55:17.163 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-20T15:55:17.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-20T15:55:17.168 INFO:tasks.ceph.osd.5:Started 2024-04-20T15:55:17.168 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-20T15:55:17.168 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-20T15:55:17.175 INFO:tasks.ceph.osd.6:Started 2024-04-20T15:55:17.175 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-20T15:55:17.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-20T15:55:17.177 INFO:tasks.ceph.osd.7:Started 2024-04-20T15:55:17.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T15:55:17.273 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.272+0000 7f04da0891c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T15:55:17.344 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.344+0000 7f04da0891c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T15:55:17.400 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:17.400 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"daea1877-30c6-40c4-b0a7-4fd11c4a648b","created":"2024-04-20T15:55:14.702765+0000","modified":"2024-04-20T15:55:17.134563+0000","last_up_change":"0.000000","last_in_change":"2024-04-20T15:55:17.134563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ea8322c2-2529-4b81-94ec-f742309b2184","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c12e91af-fc5b-47fe-b872-687eb056ef9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9758b96-7c20-49e5-b1e8-b2fb21b567b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2e9693e5-788f-40d2-a638-4bc6ee66a939","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a7519d13-2502-4a64-9af5-35082e560381","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8079627a-12e4-4244-bba2-6608e9a58860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"34517aca-45eb-442f-9270-7bda8fe5af33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d0f29ef3-eb70-4b21-a0fb-bb969904cca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T15:55:17.413 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-20T15:55:17.413 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-20T15:55:17.513 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T15:55:17.514+0000 7f3783c2e540 -1 Falling back to public interface 2024-04-20T15:55:17.523 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.524+0000 7f04da0891c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T15:55:17.576 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:17.576+0000 7f5ae7f071c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T15:55:17.600 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.601+0000 7f04da0891c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T15:55:17.645 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:17.646+0000 7f5ae7f071c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T15:55:17.674 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.675+0000 7f04da0891c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T15:55:17.767 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T15:55:17.767+0000 7fdb5a20e540 -1 Falling back to public interface 2024-04-20T15:55:17.770 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T15:55:17.770+0000 7fcb2cb0b540 -1 Falling back to public interface 2024-04-20T15:55:17.775 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T15:55:17.776+0000 7fad7fd42540 -1 Falling back to public interface 2024-04-20T15:55:17.778 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:17.778+0000 7f5ae7f071c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T15:55:17.779 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T15:55:17.779+0000 7f7c91ef3540 -1 Falling back to public interface 2024-04-20T15:55:17.779 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T15:55:17.780+0000 7fad6cfa0540 -1 Falling back to public interface 2024-04-20T15:55:17.781 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.782+0000 7f04da0891c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T15:55:17.784 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T15:55:17.785+0000 7f471705f540 -1 Falling back to public interface 2024-04-20T15:55:17.800 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T15:55:17.800+0000 7f5502271540 -1 Falling back to public interface 2024-04-20T15:55:17.850 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:17.851+0000 7f5ae7f071c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T15:55:17.959 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:17.959+0000 7f04da0891c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T15:55:18.026 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.026+0000 7f5ae7f071c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T15:55:18.036 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:18.037+0000 7f04da0891c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T15:55:18.104 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.104+0000 7f5ae7f071c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T15:55:18.178 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.178+0000 7f5ae7f071c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T15:55:18.217 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-20T15:55:18.267 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.268+0000 7f5ae7f071c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T15:55:18.323 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:18.324+0000 7f04da0891c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T15:55:18.395 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:18.396+0000 7f04da0891c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T15:55:18.403 INFO:teuthology.misc.health.smithi071.stdout: 2024-04-20T15:55:18.403 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"daea1877-30c6-40c4-b0a7-4fd11c4a648b","created":"2024-04-20T15:55:14.702765+0000","modified":"2024-04-20T15:55:17.134563+0000","last_up_change":"0.000000","last_in_change":"2024-04-20T15:55:17.134563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ea8322c2-2529-4b81-94ec-f742309b2184","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c12e91af-fc5b-47fe-b872-687eb056ef9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9758b96-7c20-49e5-b1e8-b2fb21b567b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2e9693e5-788f-40d2-a638-4bc6ee66a939","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a7519d13-2502-4a64-9af5-35082e560381","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8079627a-12e4-4244-bba2-6608e9a58860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"34517aca-45eb-442f-9270-7bda8fe5af33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d0f29ef3-eb70-4b21-a0fb-bb969904cca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T15:55:18.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-20T15:55:18.445 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.445+0000 7f5ae7f071c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T15:55:18.522 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.523+0000 7f5ae7f071c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T15:55:18.807 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.808+0000 7f5ae7f071c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T15:55:18.876 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:18.877+0000 7f5ae7f071c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T15:55:18.898 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:18.899+0000 7f04da0891c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T15:55:19.206 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:19.207+0000 7f04da0891c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T15:55:19.345 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:19.345+0000 7f04da0891c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T15:55:19.377 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:19.377+0000 7f5ae7f071c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T15:55:19.695 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:19.695+0000 7f04da0891c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T15:55:19.712 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:19.712+0000 7f5ae7f071c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T15:55:19.837 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:19.838+0000 7f04da0891c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T15:55:19.851 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:19.851+0000 7f5ae7f071c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T15:55:19.986 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:19.987+0000 7f04da0891c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T15:55:20.086 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:20.087+0000 7f04da0891c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T15:55:20.210 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:20.211+0000 7f5ae7f071c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T15:55:20.363 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:20.363+0000 7f5ae7f071c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T15:55:20.521 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:20.521+0000 7f5ae7f071c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T15:55:20.656 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-04-20T15:55:20.656+0000 7f5ae7f071c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T15:55:21.028 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T15:55:21.028+0000 7f3783c2e540 -1 osd.2 0 log_to_monitors true 2024-04-20T15:55:21.113 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:21.113+0000 7f049e78d700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-20T15:55:21.114 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T15:55:21.113+0000 7f049e78d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-20T15:55:21.236 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T15:55:21.236+0000 7fcb2cb0b540 -1 osd.5 0 log_to_monitors true 2024-04-20T15:55:21.278 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T15:55:21.278+0000 7f7c91ef3540 -1 osd.6 0 log_to_monitors true 2024-04-20T15:55:21.284 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T15:55:21.285+0000 7fdb5a20e540 -1 osd.4 0 log_to_monitors true 2024-04-20T15:55:21.292 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T15:55:21.293+0000 7fad6cfa0540 -1 osd.3 0 log_to_monitors true 2024-04-20T15:55:21.306 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T15:55:21.307+0000 7f471705f540 -1 osd.0 0 log_to_monitors true 2024-04-20T15:55:21.307 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T15:55:21.308+0000 7f5502271540 -1 osd.7 0 log_to_monitors true 2024-04-20T15:55:21.317 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T15:55:21.316+0000 7fad7fd42540 -1 osd.1 0 log_to_monitors true 2024-04-20T15:55:23.021 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T15:55:23.021+0000 7f377a833700 -1 osd.2 0 waiting for initial osdmap 2024-04-20T15:55:23.024 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T15:55:23.025+0000 7f377362a700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:24.839 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T15:55:24.840+0000 7fad7814a700 -1 osd.1 0 waiting for initial osdmap 2024-04-20T15:55:24.844 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T15:55:24.844+0000 7fad6f73e700 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:24.853 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T15:55:24.854+0000 7f470dc64700 -1 osd.0 0 waiting for initial osdmap 2024-04-20T15:55:24.856 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T15:55:24.857+0000 7fad63ba5700 -1 osd.3 0 waiting for initial osdmap 2024-04-20T15:55:24.857 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T15:55:24.858+0000 7f4706a5b700 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:24.862 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T15:55:24.863+0000 7fad5c99c700 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:25.154 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T15:55:25.155+0000 7fcb23710700 -1 osd.5 0 waiting for initial osdmap 2024-04-20T15:55:25.160 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T15:55:25.160+0000 7fcb1c507700 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:25.216 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T15:55:25.216+0000 7f54f8e76700 -1 osd.7 0 waiting for initial osdmap 2024-04-20T15:55:25.220 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T15:55:25.221+0000 7f54f1c6d700 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:25.223 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-20T15:55:25.226 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T15:55:25.227+0000 7f7c8a2fb700 -1 osd.6 0 waiting for initial osdmap 2024-04-20T15:55:25.230 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T15:55:25.230+0000 7fdb50e13700 -1 osd.4 0 waiting for initial osdmap 2024-04-20T15:55:25.231 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T15:55:25.231+0000 7f7c818ef700 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:25.235 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T15:55:25.235+0000 7fdb49c0a700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T15:55:25.528 INFO:teuthology.misc.health.smithi071.stdout: 2024-04-20T15:55:25.528 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":14,"fsid":"daea1877-30c6-40c4-b0a7-4fd11c4a648b","created":"2024-04-20T15:55:14.702765+0000","modified":"2024-04-20T15:55:24.330141+0000","last_up_change":"2024-04-20T15:55:23.163438+0000","last_in_change":"2024-04-20T15:55:17.134563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ea8322c2-2529-4b81-94ec-f742309b2184","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c12e91af-fc5b-47fe-b872-687eb056ef9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9758b96-7c20-49e5-b1e8-b2fb21b567b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":44030},{"type":"v1","addr":"172.21.15.71:6801","nonce":44030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":44030},{"type":"v1","addr":"172.21.15.71:6803","nonce":44030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":44030},{"type":"v1","addr":"172.21.15.71:6807","nonce":44030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":44030},{"type":"v1","addr":"172.21.15.71:6805","nonce":44030}]},"public_addr":"172.21.15.71:6801/44030","cluster_addr":"172.21.15.71:6803/44030","heartbeat_back_addr":"172.21.15.71:6807/44030","heartbeat_front_addr":"172.21.15.71:6805/44030","state":["exists","up"]},{"osd":3,"uuid":"2e9693e5-788f-40d2-a638-4bc6ee66a939","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a7519d13-2502-4a64-9af5-35082e560381","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8079627a-12e4-4244-bba2-6608e9a58860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"34517aca-45eb-442f-9270-7bda8fe5af33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d0f29ef3-eb70-4b21-a0fb-bb969904cca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.041089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T15:55:25.541 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-04-20T15:55:32.348 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-20T15:55:32.664 INFO:teuthology.misc.health.smithi071.stdout: 2024-04-20T15:55:32.664 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":18,"fsid":"daea1877-30c6-40c4-b0a7-4fd11c4a648b","created":"2024-04-20T15:55:14.702765+0000","modified":"2024-04-20T15:55:28.866037+0000","last_up_change":"2024-04-20T15:55:25.857789+0000","last_in_change":"2024-04-20T15:55:17.134563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T15:55:26.148529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ea8322c2-2529-4b81-94ec-f742309b2184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":44032},{"type":"v1","addr":"172.21.15.71:6825","nonce":44032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":44032},{"type":"v1","addr":"172.21.15.71:6827","nonce":44032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":44032},{"type":"v1","addr":"172.21.15.71:6831","nonce":44032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":44032},{"type":"v1","addr":"172.21.15.71:6829","nonce":44032}]},"public_addr":"172.21.15.71:6825/44032","cluster_addr":"172.21.15.71:6827/44032","heartbeat_back_addr":"172.21.15.71:6831/44032","heartbeat_front_addr":"172.21.15.71:6829/44032","state":["exists","up"]},{"osd":1,"uuid":"c12e91af-fc5b-47fe-b872-687eb056ef9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":44031},{"type":"v1","addr":"172.21.15.71:6809","nonce":44031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":44031},{"type":"v1","addr":"172.21.15.71:6811","nonce":44031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":44031},{"type":"v1","addr":"172.21.15.71:6815","nonce":44031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":44031},{"type":"v1","addr":"172.21.15.71:6813","nonce":44031}]},"public_addr":"172.21.15.71:6809/44031","cluster_addr":"172.21.15.71:6811/44031","heartbeat_back_addr":"172.21.15.71:6815/44031","heartbeat_front_addr":"172.21.15.71:6813/44031","state":["exists","up"]},{"osd":2,"uuid":"e9758b96-7c20-49e5-b1e8-b2fb21b567b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":44030},{"type":"v1","addr":"172.21.15.71:6801","nonce":44030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":44030},{"type":"v1","addr":"172.21.15.71:6803","nonce":44030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":44030},{"type":"v1","addr":"172.21.15.71:6807","nonce":44030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":44030},{"type":"v1","addr":"172.21.15.71:6805","nonce":44030}]},"public_addr":"172.21.15.71:6801/44030","cluster_addr":"172.21.15.71:6803/44030","heartbeat_back_addr":"172.21.15.71:6807/44030","heartbeat_front_addr":"172.21.15.71:6805/44030","state":["exists","up"]},{"osd":3,"uuid":"2e9693e5-788f-40d2-a638-4bc6ee66a939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":44033},{"type":"v1","addr":"172.21.15.71:6817","nonce":44033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":44033},{"type":"v1","addr":"172.21.15.71:6819","nonce":44033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":44033},{"type":"v1","addr":"172.21.15.71:6823","nonce":44033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":44033},{"type":"v1","addr":"172.21.15.71:6821","nonce":44033}]},"public_addr":"172.21.15.71:6817/44033","cluster_addr":"172.21.15.71:6819/44033","heartbeat_back_addr":"172.21.15.71:6823/44033","heartbeat_front_addr":"172.21.15.71:6821/44033","state":["exists","up"]},{"osd":4,"uuid":"a7519d13-2502-4a64-9af5-35082e560381","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":43957},{"type":"v1","addr":"172.21.15.119:6801","nonce":43957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":43957},{"type":"v1","addr":"172.21.15.119:6803","nonce":43957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":43957},{"type":"v1","addr":"172.21.15.119:6807","nonce":43957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":43957},{"type":"v1","addr":"172.21.15.119:6805","nonce":43957}]},"public_addr":"172.21.15.119:6801/43957","cluster_addr":"172.21.15.119:6803/43957","heartbeat_back_addr":"172.21.15.119:6807/43957","heartbeat_front_addr":"172.21.15.119:6805/43957","state":["exists","up"]},{"osd":5,"uuid":"8079627a-12e4-4244-bba2-6608e9a58860","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":43958},{"type":"v1","addr":"172.21.15.119:6809","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":43958},{"type":"v1","addr":"172.21.15.119:6811","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":43958},{"type":"v1","addr":"172.21.15.119:6815","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":43958},{"type":"v1","addr":"172.21.15.119:6813","nonce":43958}]},"public_addr":"172.21.15.119:6809/43958","cluster_addr":"172.21.15.119:6811/43958","heartbeat_back_addr":"172.21.15.119:6815/43958","heartbeat_front_addr":"172.21.15.119:6813/43958","state":["exists","up"]},{"osd":6,"uuid":"34517aca-45eb-442f-9270-7bda8fe5af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":43961},{"type":"v1","addr":"172.21.15.119:6817","nonce":43961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":43961},{"type":"v1","addr":"172.21.15.119:6819","nonce":43961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":43961},{"type":"v1","addr":"172.21.15.119:6823","nonce":43961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":43961},{"type":"v1","addr":"172.21.15.119:6821","nonce":43961}]},"public_addr":"172.21.15.119:6817/43961","cluster_addr":"172.21.15.119:6819/43961","heartbeat_back_addr":"172.21.15.119:6823/43961","heartbeat_front_addr":"172.21.15.119:6821/43961","state":["exists","up"]},{"osd":7,"uuid":"d0f29ef3-eb70-4b21-a0fb-bb969904cca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":44002},{"type":"v1","addr":"172.21.15.119:6825","nonce":44002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":44002},{"type":"v1","addr":"172.21.15.119:6827","nonce":44002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":44002},{"type":"v1","addr":"172.21.15.119:6831","nonce":44002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":44002},{"type":"v1","addr":"172.21.15.119:6829","nonce":44002}]},"public_addr":"172.21.15.119:6825/44002","cluster_addr":"172.21.15.119:6827/44002","heartbeat_back_addr":"172.21.15.119:6831/44002","heartbeat_front_addr":"172.21.15.119:6829/44002","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.286744+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.314779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.041089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.259773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.246701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.238370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.231235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.277875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T15:55:32.676 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-20T15:55:32.676 INFO:tasks.ceph:Creating RBD pool 2024-04-20T15:55:32.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-20T15:55:33.169 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'rbd' created 2024-04-20T15:55:33.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-20T15:55:34.179 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-20T15:55:34.192 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-20T15:55:34.192 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-20T15:55:34.193 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-20T15:55:34.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-20T15:55:34.193 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-20T15:55:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:34.581 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":43731},{"type":"v1","addr":"172.21.15.71:6833","nonce":43731}]},"active_addr":"172.21.15.71:6833/43731","active_change":"2024-04-20T15:55:20.101006+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":517856870}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3719968226}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":75421428}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2765150343}]}]} 2024-04-20T15:55:34.586 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-20T15:55:34.586 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-20T15:55:34.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T15:55:34.898 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:34.898 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":20,"fsid":"daea1877-30c6-40c4-b0a7-4fd11c4a648b","created":"2024-04-20T15:55:14.702765+0000","modified":"2024-04-20T15:55:34.165568+0000","last_up_change":"2024-04-20T15:55:25.857789+0000","last_in_change":"2024-04-20T15:55:17.134563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T15:55:26.148529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T15:55:33.004580+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ea8322c2-2529-4b81-94ec-f742309b2184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":44032},{"type":"v1","addr":"172.21.15.71:6825","nonce":44032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":44032},{"type":"v1","addr":"172.21.15.71:6827","nonce":44032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":44032},{"type":"v1","addr":"172.21.15.71:6831","nonce":44032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":44032},{"type":"v1","addr":"172.21.15.71:6829","nonce":44032}]},"public_addr":"172.21.15.71:6825/44032","cluster_addr":"172.21.15.71:6827/44032","heartbeat_back_addr":"172.21.15.71:6831/44032","heartbeat_front_addr":"172.21.15.71:6829/44032","state":["exists","up"]},{"osd":1,"uuid":"c12e91af-fc5b-47fe-b872-687eb056ef9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":44031},{"type":"v1","addr":"172.21.15.71:6809","nonce":44031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":44031},{"type":"v1","addr":"172.21.15.71:6811","nonce":44031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":44031},{"type":"v1","addr":"172.21.15.71:6815","nonce":44031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":44031},{"type":"v1","addr":"172.21.15.71:6813","nonce":44031}]},"public_addr":"172.21.15.71:6809/44031","cluster_addr":"172.21.15.71:6811/44031","heartbeat_back_addr":"172.21.15.71:6815/44031","heartbeat_front_addr":"172.21.15.71:6813/44031","state":["exists","up"]},{"osd":2,"uuid":"e9758b96-7c20-49e5-b1e8-b2fb21b567b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":44030},{"type":"v1","addr":"172.21.15.71:6801","nonce":44030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":44030},{"type":"v1","addr":"172.21.15.71:6803","nonce":44030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":44030},{"type":"v1","addr":"172.21.15.71:6807","nonce":44030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":44030},{"type":"v1","addr":"172.21.15.71:6805","nonce":44030}]},"public_addr":"172.21.15.71:6801/44030","cluster_addr":"172.21.15.71:6803/44030","heartbeat_back_addr":"172.21.15.71:6807/44030","heartbeat_front_addr":"172.21.15.71:6805/44030","state":["exists","up"]},{"osd":3,"uuid":"2e9693e5-788f-40d2-a638-4bc6ee66a939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":44033},{"type":"v1","addr":"172.21.15.71:6817","nonce":44033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":44033},{"type":"v1","addr":"172.21.15.71:6819","nonce":44033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":44033},{"type":"v1","addr":"172.21.15.71:6823","nonce":44033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":44033},{"type":"v1","addr":"172.21.15.71:6821","nonce":44033}]},"public_addr":"172.21.15.71:6817/44033","cluster_addr":"172.21.15.71:6819/44033","heartbeat_back_addr":"172.21.15.71:6823/44033","heartbeat_front_addr":"172.21.15.71:6821/44033","state":["exists","up"]},{"osd":4,"uuid":"a7519d13-2502-4a64-9af5-35082e560381","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":43957},{"type":"v1","addr":"172.21.15.119:6801","nonce":43957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":43957},{"type":"v1","addr":"172.21.15.119:6803","nonce":43957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":43957},{"type":"v1","addr":"172.21.15.119:6807","nonce":43957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":43957},{"type":"v1","addr":"172.21.15.119:6805","nonce":43957}]},"public_addr":"172.21.15.119:6801/43957","cluster_addr":"172.21.15.119:6803/43957","heartbeat_back_addr":"172.21.15.119:6807/43957","heartbeat_front_addr":"172.21.15.119:6805/43957","state":["exists","up"]},{"osd":5,"uuid":"8079627a-12e4-4244-bba2-6608e9a58860","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":43958},{"type":"v1","addr":"172.21.15.119:6809","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":43958},{"type":"v1","addr":"172.21.15.119:6811","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":43958},{"type":"v1","addr":"172.21.15.119:6815","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":43958},{"type":"v1","addr":"172.21.15.119:6813","nonce":43958}]},"public_addr":"172.21.15.119:6809/43958","cluster_addr":"172.21.15.119:6811/43958","heartbeat_back_addr":"172.21.15.119:6815/43958","heartbeat_front_addr":"172.21.15.119:6813/43958","state":["exists","up"]},{"osd":6,"uuid":"34517aca-45eb-442f-9270-7bda8fe5af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":43961},{"type":"v1","addr":"172.21.15.119:6817","nonce":43961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":43961},{"type":"v1","addr":"172.21.15.119:6819","nonce":43961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":43961},{"type":"v1","addr":"172.21.15.119:6823","nonce":43961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":43961},{"type":"v1","addr":"172.21.15.119:6821","nonce":43961}]},"public_addr":"172.21.15.119:6817/43961","cluster_addr":"172.21.15.119:6819/43961","heartbeat_back_addr":"172.21.15.119:6823/43961","heartbeat_front_addr":"172.21.15.119:6821/43961","state":["exists","up"]},{"osd":7,"uuid":"d0f29ef3-eb70-4b21-a0fb-bb969904cca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":44002},{"type":"v1","addr":"172.21.15.119:6825","nonce":44002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":44002},{"type":"v1","addr":"172.21.15.119:6827","nonce":44002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":44002},{"type":"v1","addr":"172.21.15.119:6831","nonce":44002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":44002},{"type":"v1","addr":"172.21.15.119:6829","nonce":44002}]},"public_addr":"172.21.15.119:6825/44002","cluster_addr":"172.21.15.119:6827/44002","heartbeat_back_addr":"172.21.15.119:6831/44002","heartbeat_front_addr":"172.21.15.119:6829/44002","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.286744+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.314779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.041089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.259773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.246701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.238370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.231235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.277875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T15:55:34.912 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-20T15:55:34.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T15:55:35.229 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:35.229 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"fsid":"daea1877-30c6-40c4-b0a7-4fd11c4a648b","created":"2024-04-20T15:55:14.702765+0000","modified":"2024-04-20T15:55:35.170675+0000","last_up_change":"2024-04-20T15:55:25.857789+0000","last_in_change":"2024-04-20T15:55:17.134563+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T15:55:26.148529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T15:55:33.004580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ea8322c2-2529-4b81-94ec-f742309b2184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":44032},{"type":"v1","addr":"172.21.15.71:6825","nonce":44032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":44032},{"type":"v1","addr":"172.21.15.71:6827","nonce":44032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":44032},{"type":"v1","addr":"172.21.15.71:6831","nonce":44032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":44032},{"type":"v1","addr":"172.21.15.71:6829","nonce":44032}]},"public_addr":"172.21.15.71:6825/44032","cluster_addr":"172.21.15.71:6827/44032","heartbeat_back_addr":"172.21.15.71:6831/44032","heartbeat_front_addr":"172.21.15.71:6829/44032","state":["exists","up"]},{"osd":1,"uuid":"c12e91af-fc5b-47fe-b872-687eb056ef9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":44031},{"type":"v1","addr":"172.21.15.71:6809","nonce":44031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":44031},{"type":"v1","addr":"172.21.15.71:6811","nonce":44031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":44031},{"type":"v1","addr":"172.21.15.71:6815","nonce":44031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":44031},{"type":"v1","addr":"172.21.15.71:6813","nonce":44031}]},"public_addr":"172.21.15.71:6809/44031","cluster_addr":"172.21.15.71:6811/44031","heartbeat_back_addr":"172.21.15.71:6815/44031","heartbeat_front_addr":"172.21.15.71:6813/44031","state":["exists","up"]},{"osd":2,"uuid":"e9758b96-7c20-49e5-b1e8-b2fb21b567b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":44030},{"type":"v1","addr":"172.21.15.71:6801","nonce":44030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":44030},{"type":"v1","addr":"172.21.15.71:6803","nonce":44030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":44030},{"type":"v1","addr":"172.21.15.71:6807","nonce":44030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":44030},{"type":"v1","addr":"172.21.15.71:6805","nonce":44030}]},"public_addr":"172.21.15.71:6801/44030","cluster_addr":"172.21.15.71:6803/44030","heartbeat_back_addr":"172.21.15.71:6807/44030","heartbeat_front_addr":"172.21.15.71:6805/44030","state":["exists","up"]},{"osd":3,"uuid":"2e9693e5-788f-40d2-a638-4bc6ee66a939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":44033},{"type":"v1","addr":"172.21.15.71:6817","nonce":44033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":44033},{"type":"v1","addr":"172.21.15.71:6819","nonce":44033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":44033},{"type":"v1","addr":"172.21.15.71:6823","nonce":44033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":44033},{"type":"v1","addr":"172.21.15.71:6821","nonce":44033}]},"public_addr":"172.21.15.71:6817/44033","cluster_addr":"172.21.15.71:6819/44033","heartbeat_back_addr":"172.21.15.71:6823/44033","heartbeat_front_addr":"172.21.15.71:6821/44033","state":["exists","up"]},{"osd":4,"uuid":"a7519d13-2502-4a64-9af5-35082e560381","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":43957},{"type":"v1","addr":"172.21.15.119:6801","nonce":43957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":43957},{"type":"v1","addr":"172.21.15.119:6803","nonce":43957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":43957},{"type":"v1","addr":"172.21.15.119:6807","nonce":43957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":43957},{"type":"v1","addr":"172.21.15.119:6805","nonce":43957}]},"public_addr":"172.21.15.119:6801/43957","cluster_addr":"172.21.15.119:6803/43957","heartbeat_back_addr":"172.21.15.119:6807/43957","heartbeat_front_addr":"172.21.15.119:6805/43957","state":["exists","up"]},{"osd":5,"uuid":"8079627a-12e4-4244-bba2-6608e9a58860","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":43958},{"type":"v1","addr":"172.21.15.119:6809","nonce":43958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":43958},{"type":"v1","addr":"172.21.15.119:6811","nonce":43958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":43958},{"type":"v1","addr":"172.21.15.119:6815","nonce":43958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":43958},{"type":"v1","addr":"172.21.15.119:6813","nonce":43958}]},"public_addr":"172.21.15.119:6809/43958","cluster_addr":"172.21.15.119:6811/43958","heartbeat_back_addr":"172.21.15.119:6815/43958","heartbeat_front_addr":"172.21.15.119:6813/43958","state":["exists","up"]},{"osd":6,"uuid":"34517aca-45eb-442f-9270-7bda8fe5af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":43961},{"type":"v1","addr":"172.21.15.119:6817","nonce":43961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":43961},{"type":"v1","addr":"172.21.15.119:6819","nonce":43961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":43961},{"type":"v1","addr":"172.21.15.119:6823","nonce":43961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":43961},{"type":"v1","addr":"172.21.15.119:6821","nonce":43961}]},"public_addr":"172.21.15.119:6817/43961","cluster_addr":"172.21.15.119:6819/43961","heartbeat_back_addr":"172.21.15.119:6823/43961","heartbeat_front_addr":"172.21.15.119:6821/43961","state":["exists","up"]},{"osd":7,"uuid":"d0f29ef3-eb70-4b21-a0fb-bb969904cca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":44002},{"type":"v1","addr":"172.21.15.119:6825","nonce":44002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":44002},{"type":"v1","addr":"172.21.15.119:6827","nonce":44002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":44002},{"type":"v1","addr":"172.21.15.119:6831","nonce":44002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":44002},{"type":"v1","addr":"172.21.15.119:6829","nonce":44002}]},"public_addr":"172.21.15.119:6825/44002","cluster_addr":"172.21.15.119:6827/44002","heartbeat_back_addr":"172.21.15.119:6831/44002","heartbeat_front_addr":"172.21.15.119:6829/44002","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.286744+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.314779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.041089+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.259773+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.246701+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.238370+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.231235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T15:55:22.277875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T15:55:35.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-20T15:55:35.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-20T15:55:35.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-20T15:55:35.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-20T15:55:35.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-20T15:55:35.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-20T15:55:35.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-20T15:55:35.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-20T15:55:35.517 INFO:teuthology.orchestra.run.smithi071.stdout:55834574853 2024-04-20T15:55:35.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-20T15:55:35.519 INFO:teuthology.orchestra.run.smithi071.stdout:64424509446 2024-04-20T15:55:35.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-20T15:55:35.533 INFO:teuthology.orchestra.run.smithi071.stdout:64424509444 2024-04-20T15:55:35.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-20T15:55:35.534 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:35.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-20T15:55:35.541 INFO:teuthology.orchestra.run.smithi071.stdout:64424509446 2024-04-20T15:55:35.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-20T15:55:35.542 INFO:teuthology.orchestra.run.smithi071.stdout:64424509444 2024-04-20T15:55:35.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-20T15:55:35.544 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:35.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-20T15:55:35.555 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:35.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-20T15:55:36.135 INFO:teuthology.orchestra.run.smithi071.stdout:55834574852 2024-04-20T15:55:36.138 INFO:teuthology.orchestra.run.smithi071.stdout:64424509443 2024-04-20T15:55:36.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2024-04-20T15:55:36.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.3 2024-04-20T15:55:36.175 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:36.175 INFO:teuthology.orchestra.run.smithi071.stdout:64424509444 2024-04-20T15:55:36.181 INFO:teuthology.orchestra.run.smithi071.stdout:64424509446 2024-04-20T15:55:36.184 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:36.193 INFO:teuthology.orchestra.run.smithi071.stdout:64424509444 2024-04-20T15:55:36.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.1 2024-04-20T15:55:36.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2024-04-20T15:55:36.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.5 2024-04-20T15:55:36.208 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:36.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.7 2024-04-20T15:55:36.209 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:36.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.0 2024-04-20T15:55:36.219 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:36.226 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:36.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.4 2024-04-20T15:55:36.241 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:37.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-20T15:55:37.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-20T15:55:37.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-20T15:55:37.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-20T15:55:37.521 INFO:teuthology.orchestra.run.smithi071.stdout:55834574853 2024-04-20T15:55:37.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2024-04-20T15:55:37.536 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:37.581 INFO:teuthology.orchestra.run.smithi071.stdout:64424509446 2024-04-20T15:55:37.598 INFO:teuthology.orchestra.run.smithi071.stdout:64424509444 2024-04-20T15:55:37.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.6 2024-04-20T15:55:37.601 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:37.606 INFO:teuthology.orchestra.run.smithi071.stdout:64424509445 2024-04-20T15:55:37.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.3 2024-04-20T15:55:37.612 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:37.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.1 2024-04-20T15:55:37.620 DEBUG:teuthology.parallel:result is None 2024-04-20T15:55:37.620 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-20T15:55:37.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T15:55:37.970 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:37.971 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-04-20T15:55:37.982 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-20T15:55:36.112697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2381612,"kb_used_data":2732,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752593108,"statfs":{"total":773094113280,"available":770655342592,"internally_reserved":0,"allocated":2797568,"data_stored":1699208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004580"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.179230+0000","last_change":"2024-04-20T15:55:34.178040+0000","last_active":"2024-04-20T15:55:34.179230+0000","last_peered":"2024-04-20T15:55:34.179230+0000","last_clean":"2024-04-20T15:55:34.179230+0000","last_became_active":"2024-04-20T15:55:34.177759+0000","last_became_peered":"2024-04-20T15:55:34.177759+0000","last_unstale":"2024-04-20T15:55:34.179230+0000","last_undegraded":"2024-04-20T15:55:34.179230+0000","last_fullsized":"2024-04-20T15:55:34.179230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T01:21:09.708761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.183544+0000","last_change":"2024-04-20T15:55:34.178083+0000","last_active":"2024-04-20T15:55:34.183544+0000","last_peered":"2024-04-20T15:55:34.183544+0000","last_clean":"2024-04-20T15:55:34.183544+0000","last_became_active":"2024-04-20T15:55:34.177757+0000","last_became_peered":"2024-04-20T15:55:34.177757+0000","last_unstale":"2024-04-20T15:55:34.183544+0000","last_undegraded":"2024-04-20T15:55:34.183544+0000","last_fullsized":"2024-04-20T15:55:34.183544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T20:23:14.437574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.179550+0000","last_change":"2024-04-20T15:55:34.179550+0000","last_active":"2024-04-20T15:55:34.179550+0000","last_peered":"2024-04-20T15:55:34.179550+0000","last_clean":"2024-04-20T15:55:34.179550+0000","last_became_active":"2024-04-20T15:55:34.179318+0000","last_became_peered":"2024-04-20T15:55:34.179318+0000","last_unstale":"2024-04-20T15:55:34.179550+0000","last_undegraded":"2024-04-20T15:55:34.179550+0000","last_fullsized":"2024-04-20T15:55:34.179550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T03:24:01.605412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.179077+0000","last_change":"2024-04-20T15:55:34.179077+0000","last_active":"2024-04-20T15:55:34.179077+0000","last_peered":"2024-04-20T15:55:34.179077+0000","last_clean":"2024-04-20T15:55:34.179077+0000","last_became_active":"2024-04-20T15:55:34.178802+0000","last_became_peered":"2024-04-20T15:55:34.178802+0000","last_unstale":"2024-04-20T15:55:34.179077+0000","last_undegraded":"2024-04-20T15:55:34.179077+0000","last_fullsized":"2024-04-20T15:55:34.179077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T20:23:14.437574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T15:55:33.171501+0000","last_change":"2024-04-20T15:55:27.870850+0000","last_active":"2024-04-20T15:55:33.171501+0000","last_peered":"2024-04-20T15:55:33.171501+0000","last_clean":"2024-04-20T15:55:33.171501+0000","last_became_active":"2024-04-20T15:55:27.870598+0000","last_became_peered":"2024-04-20T15:55:27.870598+0000","last_unstale":"2024-04-20T15:55:33.171501+0000","last_undegraded":"2024-04-20T15:55:33.171501+0000","last_fullsized":"2024-04-20T15:55:33.171501+0000","mapping_epoch":16,"log_start":"17'100","ondisk_log_start":"17'100","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:26.859472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:26.859472+0000","last_clean_scrub_stamp":"2024-04-20T15:55:26.859472+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T19:58:55.612467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.181432+0000","last_change":"2024-04-20T15:55:34.181432+0000","last_active":"2024-04-20T15:55:34.181432+0000","last_peered":"2024-04-20T15:55:34.181432+0000","last_clean":"2024-04-20T15:55:34.181432+0000","last_became_active":"2024-04-20T15:55:34.181198+0000","last_became_peered":"2024-04-20T15:55:34.181198+0000","last_unstale":"2024-04-20T15:55:34.181432+0000","last_undegraded":"2024-04-20T15:55:34.181432+0000","last_fullsized":"2024-04-20T15:55:34.181432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T03:24:01.605412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.184349+0000","last_change":"2024-04-20T15:55:34.184349+0000","last_active":"2024-04-20T15:55:34.184349+0000","last_peered":"2024-04-20T15:55:34.184349+0000","last_clean":"2024-04-20T15:55:34.184349+0000","last_became_active":"2024-04-20T15:55:34.184062+0000","last_became_peered":"2024-04-20T15:55:34.184062+0000","last_unstale":"2024-04-20T15:55:34.184349+0000","last_undegraded":"2024-04-20T15:55:34.184349+0000","last_fullsized":"2024-04-20T15:55:34.184349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T21:45:37.131045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.184119+0000","last_change":"2024-04-20T15:55:34.184119+0000","last_active":"2024-04-20T15:55:34.184119+0000","last_peered":"2024-04-20T15:55:34.184119+0000","last_clean":"2024-04-20T15:55:34.184119+0000","last_became_active":"2024-04-20T15:55:34.183876+0000","last_became_peered":"2024-04-20T15:55:34.183876+0000","last_unstale":"2024-04-20T15:55:34.184119+0000","last_undegraded":"2024-04-20T15:55:34.184119+0000","last_fullsized":"2024-04-20T15:55:34.184119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T20:23:14.437574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.184478+0000","last_change":"2024-04-20T15:55:34.179603+0000","last_active":"2024-04-20T15:55:34.184478+0000","last_peered":"2024-04-20T15:55:34.184478+0000","last_clean":"2024-04-20T15:55:34.184478+0000","last_became_active":"2024-04-20T15:55:34.179248+0000","last_became_peered":"2024-04-20T15:55:34.179248+0000","last_unstale":"2024-04-20T15:55:34.184478+0000","last_undegraded":"2024-04-20T15:55:34.184478+0000","last_fullsized":"2024-04-20T15:55:34.184478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T22:34:52.854020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298132,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073708,"statfs":{"total":96636764160,"available":96331476992,"internally_reserved":0,"allocated":790528,"data_stored":653026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":68394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298148,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073692,"statfs":{"total":96636764160,"available":96331460608,"internally_reserved":0,"allocated":802816,"data_stored":658762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":68394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297548,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T15:55:37.984 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T15:55:38.296 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:38.298 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2024-04-20T15:55:38.309 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-04-20T15:55:38.113049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2381612,"kb_used_data":2732,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":752593108,"statfs":{"total":773094113280,"available":770655342592,"internally_reserved":0,"allocated":2797568,"data_stored":1699208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004931"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.179230+0000","last_change":"2024-04-20T15:55:34.178040+0000","last_active":"2024-04-20T15:55:34.179230+0000","last_peered":"2024-04-20T15:55:34.179230+0000","last_clean":"2024-04-20T15:55:34.179230+0000","last_became_active":"2024-04-20T15:55:34.177759+0000","last_became_peered":"2024-04-20T15:55:34.177759+0000","last_unstale":"2024-04-20T15:55:34.179230+0000","last_undegraded":"2024-04-20T15:55:34.179230+0000","last_fullsized":"2024-04-20T15:55:34.179230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T01:21:09.708761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.183544+0000","last_change":"2024-04-20T15:55:34.178083+0000","last_active":"2024-04-20T15:55:34.183544+0000","last_peered":"2024-04-20T15:55:34.183544+0000","last_clean":"2024-04-20T15:55:34.183544+0000","last_became_active":"2024-04-20T15:55:34.177757+0000","last_became_peered":"2024-04-20T15:55:34.177757+0000","last_unstale":"2024-04-20T15:55:34.183544+0000","last_undegraded":"2024-04-20T15:55:34.183544+0000","last_fullsized":"2024-04-20T15:55:34.183544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T20:23:14.437574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.179550+0000","last_change":"2024-04-20T15:55:34.179550+0000","last_active":"2024-04-20T15:55:34.179550+0000","last_peered":"2024-04-20T15:55:34.179550+0000","last_clean":"2024-04-20T15:55:34.179550+0000","last_became_active":"2024-04-20T15:55:34.179318+0000","last_became_peered":"2024-04-20T15:55:34.179318+0000","last_unstale":"2024-04-20T15:55:34.179550+0000","last_undegraded":"2024-04-20T15:55:34.179550+0000","last_fullsized":"2024-04-20T15:55:34.179550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T03:24:01.605412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.179077+0000","last_change":"2024-04-20T15:55:34.179077+0000","last_active":"2024-04-20T15:55:34.179077+0000","last_peered":"2024-04-20T15:55:34.179077+0000","last_clean":"2024-04-20T15:55:34.179077+0000","last_became_active":"2024-04-20T15:55:34.178802+0000","last_became_peered":"2024-04-20T15:55:34.178802+0000","last_unstale":"2024-04-20T15:55:34.179077+0000","last_undegraded":"2024-04-20T15:55:34.179077+0000","last_fullsized":"2024-04-20T15:55:34.179077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T20:23:14.437574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T15:55:33.171501+0000","last_change":"2024-04-20T15:55:27.870850+0000","last_active":"2024-04-20T15:55:33.171501+0000","last_peered":"2024-04-20T15:55:33.171501+0000","last_clean":"2024-04-20T15:55:33.171501+0000","last_became_active":"2024-04-20T15:55:27.870598+0000","last_became_peered":"2024-04-20T15:55:27.870598+0000","last_unstale":"2024-04-20T15:55:33.171501+0000","last_undegraded":"2024-04-20T15:55:33.171501+0000","last_fullsized":"2024-04-20T15:55:33.171501+0000","mapping_epoch":16,"log_start":"17'100","ondisk_log_start":"17'100","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:26.859472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:26.859472+0000","last_clean_scrub_stamp":"2024-04-20T15:55:26.859472+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T19:58:55.612467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.181432+0000","last_change":"2024-04-20T15:55:34.181432+0000","last_active":"2024-04-20T15:55:34.181432+0000","last_peered":"2024-04-20T15:55:34.181432+0000","last_clean":"2024-04-20T15:55:34.181432+0000","last_became_active":"2024-04-20T15:55:34.181198+0000","last_became_peered":"2024-04-20T15:55:34.181198+0000","last_unstale":"2024-04-20T15:55:34.181432+0000","last_undegraded":"2024-04-20T15:55:34.181432+0000","last_fullsized":"2024-04-20T15:55:34.181432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T03:24:01.605412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.184349+0000","last_change":"2024-04-20T15:55:34.184349+0000","last_active":"2024-04-20T15:55:34.184349+0000","last_peered":"2024-04-20T15:55:34.184349+0000","last_clean":"2024-04-20T15:55:34.184349+0000","last_became_active":"2024-04-20T15:55:34.184062+0000","last_became_peered":"2024-04-20T15:55:34.184062+0000","last_unstale":"2024-04-20T15:55:34.184349+0000","last_undegraded":"2024-04-20T15:55:34.184349+0000","last_fullsized":"2024-04-20T15:55:34.184349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T21:45:37.131045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.184119+0000","last_change":"2024-04-20T15:55:34.184119+0000","last_active":"2024-04-20T15:55:34.184119+0000","last_peered":"2024-04-20T15:55:34.184119+0000","last_clean":"2024-04-20T15:55:34.184119+0000","last_became_active":"2024-04-20T15:55:34.183876+0000","last_became_peered":"2024-04-20T15:55:34.183876+0000","last_unstale":"2024-04-20T15:55:34.184119+0000","last_undegraded":"2024-04-20T15:55:34.184119+0000","last_fullsized":"2024-04-20T15:55:34.184119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T20:23:14.437574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T15:55:34.184478+0000","last_change":"2024-04-20T15:55:34.179603+0000","last_active":"2024-04-20T15:55:34.184478+0000","last_peered":"2024-04-20T15:55:34.184478+0000","last_clean":"2024-04-20T15:55:34.184478+0000","last_became_active":"2024-04-20T15:55:34.179248+0000","last_became_peered":"2024-04-20T15:55:34.179248+0000","last_unstale":"2024-04-20T15:55:34.184478+0000","last_undegraded":"2024-04-20T15:55:34.184478+0000","last_fullsized":"2024-04-20T15:55:34.184478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T15:55:33.160270+0000","last_clean_scrub_stamp":"2024-04-20T15:55:33.160270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-21T22:34:52.854020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298132,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073708,"statfs":{"total":96636764160,"available":96331476992,"internally_reserved":0,"allocated":790528,"data_stored":653026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":68394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298148,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073692,"statfs":{"total":96636764160,"available":96331460608,"internally_reserved":0,"allocated":802816,"data_stored":658762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":68394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297548,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":196608,"data_stored":62658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T15:55:38.310 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-20T15:55:38.310 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-20T15:55:38.310 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-20T15:55:38.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-20T15:55:38.658 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:38.658 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-20T15:55:38.671 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-20T15:55:38.672 INFO:teuthology.run_tasks:Running task openssl_keys... 2024-04-20T15:55:38.688 INFO:teuthology.run_tasks:Running task rgw... 2024-04-20T15:55:38.700 DEBUG:tasks.rgw:config is {'client.0': None} 2024-04-20T15:55:38.700 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-04-20T15:55:38.700 INFO:tasks.rgw:Creating data pools 2024-04-20T15:55:38.700 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-04-20T15:55:38.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2024-04-20T15:55:39.205 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.data' created 2024-04-20T15:55:39.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-04-20T15:55:40.219 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-20T15:55:40.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-04-20T15:55:41.219 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'default.rgw.buckets.index' created 2024-04-20T15:55:41.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-04-20T15:55:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-20T15:55:42.246 DEBUG:tasks.rgw:Pools created 2024-04-20T15:55:42.246 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2024-04-20T15:55:42.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2024-04-20T15:55:42.246 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2024-04-20T15:55:42.362 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:55:42.363 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:55:42.369 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:42.369+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.237+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.237 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.237+0000 7f5df904b980 20 realm 2024-04-20T15:55:44.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.237+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.237+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.237+0000 7f5df904b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-20T15:55:44.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.237+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.238+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.238+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.240+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.240+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.240+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.240+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.240+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.240+0000 7f5df904b980 10 Creating default zonegroup 2024-04-20T15:55:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.241+0000 7f5df904b980 20 get_system_obj_state: rctx=0x7fff434d5760 obj=:.pools.avail state=0x556bdf27e240 s->prefetch_data=0 2024-04-20T15:55:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.241+0000 7f5df904b980 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-20T15:55:44.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.254+0000 7f5df904b980 10 WARNING: store->list_zones() returned r=-2 2024-04-20T15:55:44.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.254+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.254 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.254+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.257+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.257+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.257+0000 7f5df904b980 10 could not read realm id: (2) No such file or directory 2024-04-20T15:55:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.257+0000 7f5df904b980 10 WARNING: failed to set zone as default, r=-22 2024-04-20T15:55:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.257+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.257+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.261 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.261+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.262+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.262+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.263+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=362 2024-04-20T15:55:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.263+0000 7f5df904b980 10 Using default name default 2024-04-20T15:55:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.263+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.264+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.264+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.264+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.264+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.264+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.264+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.265+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.265+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.265+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:44.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.274+0000 7f5df904b980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:55:44.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.274+0000 7f5df904b980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:55:44.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.274+0000 7f5df904b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.274+0000 7f5df904b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.274+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.275+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.275+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.275+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.275+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.275+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.275+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 rados_obj.operate() r=0 bl.length=362 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 zone default found 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 4 Realm: () 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 4 ZoneGroup: default (3221e4dc-597a-4adc-b880-a5fd67f97b2e) 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 4 Zone: default (f76a52bc-49fc-4b9b-bb36-9008f7b8b677) 2024-04-20T15:55:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-20T15:55:44.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 zonegroup default 2024-04-20T15:55:44.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:44.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:44.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:44.276+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:46.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:46.244+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:46.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:46.244+0000 7f5df904b980 20 rados->read ofs=0 len=0 2024-04-20T15:55:46.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:46.244+0000 7f5df904b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:46.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:46.245+0000 7f5df904b980 20 started sync module instance, tier type = 2024-04-20T15:55:46.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:46.245+0000 7f5df904b980 20 started zone id=f76a52bc-49fc-4b9b-bb36-9008f7b8b677 (name=default) with tier type = 2024-04-20T15:55:48.265 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=0 2024-04-20T15:55:48.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=1 2024-04-20T15:55:48.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=2 2024-04-20T15:55:48.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=3 2024-04-20T15:55:48.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=4 2024-04-20T15:55:48.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=5 2024-04-20T15:55:48.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.265+0000 7f5df904b980 20 add_watcher() i=6 2024-04-20T15:55:48.267 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.266+0000 7f5df904b980 20 add_watcher() i=7 2024-04-20T15:55:48.267 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.266+0000 7f5df904b980 2 all 8 watchers are set, enabling cache 2024-04-20T15:55:48.267 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-20T15:55:48.267 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-20T15:55:48.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-20T15:55:48.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.267+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-20T15:55:48.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2024-04-20T15:55:48.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.268+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2024-04-20T15:55:48.271 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2024-04-20T15:55:48.272 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.269+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.273 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.270+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.271+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.275 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.272+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2024-04-20T15:55:48.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.273+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2024-04-20T15:55:48.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2024-04-20T15:55:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.274+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.275+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2024-04-20T15:55:48.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2024-04-20T15:55:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.276+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.286 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2024-04-20T15:55:48.286 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2024-04-20T15:55:48.286 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.286 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.277+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.278+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.279+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2024-04-20T15:55:48.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2024-04-20T15:55:48.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.280+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.281+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.282+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2024-04-20T15:55:48.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2024-04-20T15:55:48.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2024-04-20T15:55:48.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2024-04-20T15:55:48.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.283+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2024-04-20T15:55:48.302 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.284+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.303 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.285+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2024-04-20T15:55:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.286+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.305 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.287+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.288+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2024-04-20T15:55:48.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.289+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.290+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2024-04-20T15:55:48.309 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.291+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2024-04-20T15:55:48.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.292+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2024-04-20T15:55:48.311 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.293+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2024-04-20T15:55:48.312 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.294+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.313 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.295+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2024-04-20T15:55:48.314 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.296+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.297+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.297+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2024-04-20T15:55:48.315 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.297+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.297+0000 7f5df904b980 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2024-04-20T15:55:48.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.297+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1205 entering 2024-04-20T15:55:48.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.298+0000 7f5df904b980 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-04-20T15:55:48.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.303+0000 7f5dbd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-20T15:55:48.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.304+0000 7f5df904b980 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-20T15:55:48.316 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:48.304+0000 7f5df904b980 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-20T15:55:51.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.240+0000 7f5df904b980 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-20T15:55:51.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.240+0000 7f5df904b980 5 note: GC not initialized 2024-04-20T15:55:51.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.242+0000 7f5d1ffff700 20 reqs_thread_entry: start 2024-04-20T15:55:51.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.242+0000 7f5d1dffb700 20 reqs_thread_entry: start 2024-04-20T15:55:51.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.242+0000 7f5df904b980 20 init_complete bucket index max shards: 11 2024-04-20T15:55:51.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.243+0000 7f5df904b980 10 rgw notify: Started notification manager with: 1 workers 2024-04-20T15:55:51.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.243+0000 7f5d0ffff700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 20 15:56:21 2024 2024-04-20T15:55:51.244 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T15:55:51.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.243+0000 7f5d0f7fe700 20 reqs_thread_entry: start 2024-04-20T15:55:51.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.265+0000 7f5df904b980 20 remove_watcher() i=0 2024-04-20T15:55:51.266 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.265+0000 7f5df904b980 2 removed watcher, disabling cache 2024-04-20T15:55:51.267 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.267+0000 7f5df904b980 20 remove_watcher() i=1 2024-04-20T15:55:51.268 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.268+0000 7f5df904b980 20 remove_watcher() i=2 2024-04-20T15:55:51.272 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.272+0000 7f5df904b980 20 remove_watcher() i=3 2024-04-20T15:55:51.273 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.273+0000 7f5df904b980 20 remove_watcher() i=4 2024-04-20T15:55:51.274 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.274+0000 7f5df904b980 20 remove_watcher() i=5 2024-04-20T15:55:51.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.277+0000 7f5df904b980 20 remove_watcher() i=6 2024-04-20T15:55:51.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.278+0000 7f5df904b980 20 remove_watcher() i=7 2024-04-20T15:55:51.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.279+0000 7f5dbd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-20T15:55:51.283 INFO:teuthology.orchestra.run.smithi071.stdout:[] 2024-04-20T15:55:51.286 DEBUG:tasks.util.rgw: json result: [] 2024-04-20T15:55:51.286 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2024-04-20T15:55:51.286 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-04-20T15:55:51.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2024-04-20T15:55:51.287 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2024-04-20T15:55:51.337 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:55:51.338 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:55:51.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.344+0000 7fa0c8aaa980 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-20T15:55:51.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.344+0000 7fa0c8aaa980 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-20T15:55:51.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.349+0000 7fa034ff9700 20 reqs_thread_entry: start 2024-04-20T15:55:51.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.349+0000 7fa0c8aaa980 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-04-20T15:55:51.349 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.349+0000 7fa0c8aaa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.350+0000 7fa0c8aaa980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.350+0000 7fa0c8aaa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.350 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.350+0000 7fa0c8aaa980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.361+0000 7fa0c8aaa980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.361+0000 7fa0c8aaa980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.361+0000 7fa0c8aaa980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.361+0000 7fa0c8aaa980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.361+0000 7fa0c8aaa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.362+0000 7fa0c8aaa980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.362+0000 7fa0c8aaa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.362+0000 7fa0c8aaa980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.362+0000 7fa0c8aaa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.362+0000 7fa0c8aaa980 20 rados_obj.operate() r=0 bl.length=362 2024-04-20T15:55:51.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.362+0000 7fa0c8aaa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.363+0000 7fa0c8aaa980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2024-04-20T15:55:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "LUKEWARM", 2024-04-20T15:55:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2024-04-20T15:55:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-20T15:55:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.368 INFO:teuthology.orchestra.run.smithi071.stdout:] 2024-04-20T15:55:51.369 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2024-04-20T15:55:51.369 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-04-20T15:55:51.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2024-04-20T15:55:51.370 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2024-04-20T15:55:51.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:55:51.420 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:55:51.426 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.426+0000 7f3e84103980 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-20T15:55:51.426 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.426+0000 7f3e84103980 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-20T15:55:51.431 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.431+0000 7f3dbffff700 20 reqs_thread_entry: start 2024-04-20T15:55:51.431 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.431+0000 7f3e84103980 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-04-20T15:55:51.431 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.431+0000 7f3e84103980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.432 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.432+0000 7f3e84103980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.432 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.432+0000 7f3e84103980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.432 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.432+0000 7f3e84103980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.442+0000 7f3e84103980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:55:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.442+0000 7f3e84103980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:55:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.442+0000 7f3e84103980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:55:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.442+0000 7f3e84103980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:55:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.442+0000 7f3e84103980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.443+0000 7f3e84103980 20 rados_obj.operate() r=0 bl.length=874 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.443+0000 7f3e84103980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.443+0000 7f3e84103980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.443+0000 7f3e84103980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.443+0000 7f3e84103980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.443+0000 7f3e84103980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.444+0000 7f3e84103980 20 rados_obj.operate() r=0 bl.length=386 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "f76a52bc-49fc-4b9b-bb36-9008f7b8b677", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "default.rgw.meta:root", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "default.rgw.control", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "default.rgw.log", 2024-04-20T15:55:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "default.rgw.otp", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "LUKEWARM": { 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data" 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-20T15:55:51.451 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": "true" 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "", 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-20T15:55:51.452 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-20T15:55:51.453 DEBUG:tasks.util.rgw: json result: {'id': 'f76a52bc-49fc-4b9b-bb36-9008f7b8b677', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-20T15:55:51.453 INFO:tasks.rgw:Configuring storage class type = FROZEN 2024-04-20T15:55:51.453 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-04-20T15:55:51.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2024-04-20T15:55:51.454 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2024-04-20T15:55:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:55:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:55:51.507 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.507+0000 7faf791a5980 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-20T15:55:51.507 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.507+0000 7faf791a5980 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-20T15:55:51.512 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.511+0000 7faebd7fa700 20 reqs_thread_entry: start 2024-04-20T15:55:51.512 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.511+0000 7faf791a5980 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-04-20T15:55:51.512 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.511+0000 7faf791a5980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.512 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.512+0000 7faf791a5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.513 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.512+0000 7faf791a5980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.513 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.513+0000 7faf791a5980 20 rados_obj.operate() r=0 bl.length=941 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:51.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.523+0000 7faf791a5980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.524+0000 7faf791a5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.524+0000 7faf791a5980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.524+0000 7faf791a5980 20 rados_obj.operate() r=0 bl.length=386 2024-04-20T15:55:51.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.524+0000 7faf791a5980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.524+0000 7faf791a5980 20 rados_obj.operate() r=0 bl.length=941 2024-04-20T15:55:51.528 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2024-04-20T15:55:51.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-20T15:55:51.528 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2024-04-20T15:55:51.528 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default-placement", 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "tags": [], 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": [ 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "FROZEN", 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "LUKEWARM", 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD" 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.529 INFO:teuthology.orchestra.run.smithi071.stdout:] 2024-04-20T15:55:51.530 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2024-04-20T15:55:51.530 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-04-20T15:55:51.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2024-04-20T15:55:51.531 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2024-04-20T15:55:51.580 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:55:51.580 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:55:51.586 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.586+0000 7f7de13aa980 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-20T15:55:51.586 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.586+0000 7f7de13aa980 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-20T15:55:51.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.590+0000 7f7d1dffb700 20 reqs_thread_entry: start 2024-04-20T15:55:51.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.590+0000 7f7de13aa980 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-04-20T15:55:51.590 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.590+0000 7f7de13aa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.591+0000 7f7de13aa980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.591+0000 7f7de13aa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.591 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.591+0000 7f7de13aa980 20 rados_obj.operate() r=0 bl.length=941 2024-04-20T15:55:51.602 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 rados_obj.operate() r=0 bl.length=941 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.602+0000 7f7de13aa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.603 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.603+0000 7f7de13aa980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:55:51.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.603+0000 7f7de13aa980 20 rados->read ofs=0 len=0 2024-04-20T15:55:51.604 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:55:51.604+0000 7f7de13aa980 20 rados_obj.operate() r=0 bl.length=396 2024-04-20T15:55:51.608 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "id": "f76a52bc-49fc-4b9b-bb36-9008f7b8b677", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "name": "default", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "domain_root": "default.rgw.meta:root", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "control_pool": "default.rgw.control", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "log_pool": "default.rgw.log", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-20T15:55:51.609 INFO:teuthology.orchestra.run.smithi071.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "otp_pool": "default.rgw.otp", 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "system_key": { 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "", 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "" 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_pools": [ 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "key": "default-placement", 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "val": { 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "storage_classes": { 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "FROZEN": { 2024-04-20T15:55:51.610 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data.frozen" 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "LUKEWARM": { 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "STANDARD": { 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "data_pool": "default.rgw.buckets.data" 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "index_type": 0, 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: "inline_data": "true" 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.611 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:55:51.612 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-20T15:55:51.612 INFO:teuthology.orchestra.run.smithi071.stdout: "realm_id": "", 2024-04-20T15:55:51.612 INFO:teuthology.orchestra.run.smithi071.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-20T15:55:51.612 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-20T15:55:51.613 DEBUG:tasks.util.rgw: json result: {'id': 'f76a52bc-49fc-4b9b-bb36-9008f7b8b677', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-20T15:55:51.613 INFO:tasks.rgw:Starting rgw... 2024-04-20T15:55:51.613 INFO:tasks.rgw:rgw client.0 config is {} 2024-04-20T15:55:51.613 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-20T15:55:51.613 INFO:tasks.rgw.client.0:Restarting daemon 2024-04-20T15:55:51.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-04-20T15:55:51.616 INFO:tasks.rgw.client.0:Started 2024-04-20T15:55:51.616 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi071.front.sepia.ceph.com:80/ 2024-04-20T15:55:51.616 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:80/ 2024-04-20T15:55:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-20T15:55:51.650 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T15:55:51.650 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-04-20T15:55:51.650 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi071.front.sepia.ceph.com port 80: Connection refused 2024-04-20T15:55:52.651 DEBUG:teuthology.orchestra.run.smithi071:> curl http://smithi071.front.sepia.ceph.com:80/ 2024-04-20T15:55:52.674 INFO:teuthology.orchestra.run.smithi071.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-20T15:55:52.674 INFO:teuthology.orchestra.run.smithi071.stderr: Dload Upload Total Spent Left Speed 2024-04-20T15:55:52.676 INFO:teuthology.orchestra.run.smithi071.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-20T15:55:52.677 INFO:teuthology.orchestra.run.smithi071.stdout:anonymous 2024-04-20T15:55:52.677 INFO:teuthology.run_tasks:Running task rabbitmq... 2024-04-20T15:55:52.690 DEBUG:tasks.rabbitmq:RabbitMQ config is {'client.0': None} 2024-04-20T15:55:52.690 INFO:tasks.rabbitmq:Installing RabbitMQ... 2024-04-20T15:55:52.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install epel-release 2024-04-20T15:55:53.455 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:04:11 ago on Sat 20 Apr 2024 03:51:42 PM UTC. 2024-04-20T15:55:55.103 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:55:55.103 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout: epel-release noarch 8-19.el8 epel 25 k 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout: epel-next-release noarch 8-19.el8 epel 11 k 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-04-20T15:55:55.104 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:55.105 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 35 k 2024-04-20T15:55:55.105 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 37 k 2024-04-20T15:55:55.105 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:55:55.547 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): epel-next-release-8-19.el8.noarch.rpm 46 kB/s | 11 kB 00:00 2024-04-20T15:55:55.572 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): epel-release-8-19.el8.noarch.rpm 94 kB/s | 25 kB 00:00 2024-04-20T15:55:55.572 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:55:55.572 INFO:teuthology.orchestra.run.smithi071.stdout:Total 76 kB/s | 35 kB 00:00 2024-04-20T15:55:55.573 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:55:55.578 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:55:55.578 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:55:55.580 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:55:55.583 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:55:55.676 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:55:55.676 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : epel-release-8-19.el8.noarch 1/2 2024-04-20T15:55:55.676 INFO:teuthology.orchestra.run.smithi071.stdout:warning: /etc/yum.repos.d/epel-testing.repo created as /etc/yum.repos.d/epel-testing.repo.rpmnew 2024-04-20T15:55:55.676 INFO:teuthology.orchestra.run.smithi071.stdout:warning: /etc/yum.repos.d/epel.repo created as /etc/yum.repos.d/epel.repo.rpmnew 2024-04-20T15:55:55.676 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:55.868 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: epel-release-8-19.el8.noarch 1/2 2024-04-20T15:55:55.869 INFO:teuthology.orchestra.run.smithi071.stdout:Many EPEL packages require the CodeReady Builder (CRB) repository. 2024-04-20T15:55:55.869 INFO:teuthology.orchestra.run.smithi071.stdout:It is recommended that you run /usr/bin/crb enable to enable the CRB repository. 2024-04-20T15:55:55.869 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:56.019 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : epel-next-release-8-19.el8.noarch 2/2 2024-04-20T15:55:56.415 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: epel-next-release-8-19.el8.noarch 2/2 2024-04-20T15:55:56.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : epel-next-release-8-19.el8.noarch 1/2 2024-04-20T15:55:56.799 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : epel-release-8-19.el8.noarch 2/2 2024-04-20T15:55:56.799 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:56.799 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:55:56.799 INFO:teuthology.orchestra.run.smithi071.stdout: epel-next-release-8-19.el8.noarch epel-release-8-19.el8.noarch 2024-04-20T15:55:56.799 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:55:56.799 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:55:56.959 DEBUG:teuthology.orchestra.run.smithi071:> curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash 2024-04-20T15:55:57.349 INFO:teuthology.orchestra.run.smithi071.stdout:Detected operating system as centos/8. 2024-04-20T15:55:57.349 INFO:teuthology.orchestra.run.smithi071.stdout:Checking for curl... 2024-04-20T15:55:57.349 INFO:teuthology.orchestra.run.smithi071.stdout:Detected curl... 2024-04-20T15:55:57.349 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading repository file: https://packagecloud.io/install/repositories/rabbitmq/erlang/config_file.repo?os=centos&dist=8&source=script 2024-04-20T15:55:57.680 INFO:teuthology.orchestra.run.smithi071.stdout:done. 2024-04-20T15:55:57.680 INFO:teuthology.orchestra.run.smithi071.stdout:Installing yum-utils... 2024-04-20T15:55:59.010 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 8 - Next - 609 kB/s | 368 kB 00:00 2024-04-20T15:56:00.522 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq_erlang-source 639 B/s | 819 B 00:01 2024-04-20T15:56:00.889 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq_erlang-source 11 kB/s | 3.8 kB 00:00 2024-04-20T15:56:00.947 INFO:teuthology.orchestra.run.smithi071.stderr:Importing GPG key 0xDF309A0B: 2024-04-20T15:56:00.947 INFO:teuthology.orchestra.run.smithi071.stderr: Userid : "https://packagecloud.io/rabbitmq/erlang (https://packagecloud.io/docs#gpg_signing) " 2024-04-20T15:56:00.948 INFO:teuthology.orchestra.run.smithi071.stderr: Fingerprint: 2EBD E413 D3CE 5D35 BCD1 5B7C 71C6 3471 DF30 9A0B 2024-04-20T15:56:00.948 INFO:teuthology.orchestra.run.smithi071.stderr: From : https://packagecloud.io/rabbitmq/erlang/gpgkey 2024-04-20T15:56:02.972 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq_erlang-source 147 B/s | 296 B 00:02 2024-04-20T15:56:03.082 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:01 ago on Sat 20 Apr 2024 03:56:02 PM UTC. 2024-04-20T15:56:05.071 INFO:teuthology.orchestra.run.smithi071.stdout:Package yum-utils-4.0.21-25.el8.noarch is already installed. 2024-04-20T15:56:05.122 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:56:05.123 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:56:05.123 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:56:05.829 INFO:teuthology.orchestra.run.smithi071.stdout:Generating yum cache for rabbitmq_erlang... 2024-04-20T15:56:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:Importing GPG key 0xDF309A0B: 2024-04-20T15:56:07.754 INFO:teuthology.orchestra.run.smithi071.stderr: Userid : "https://packagecloud.io/rabbitmq/erlang (https://packagecloud.io/docs#gpg_signing) " 2024-04-20T15:56:07.754 INFO:teuthology.orchestra.run.smithi071.stderr: Fingerprint: 2EBD E413 D3CE 5D35 BCD1 5B7C 71C6 3471 DF30 9A0B 2024-04-20T15:56:07.754 INFO:teuthology.orchestra.run.smithi071.stderr: From : https://packagecloud.io/rabbitmq/erlang/gpgkey 2024-04-20T15:56:10.519 INFO:teuthology.orchestra.run.smithi071.stdout:Generating yum cache for rabbitmq_erlang-source... 2024-04-20T15:56:12.155 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:12.155 INFO:teuthology.orchestra.run.smithi071.stdout:The repository is setup! You can now install packages. 2024-04-20T15:56:12.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install erlang 2024-04-20T15:56:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout: erlang x86_64 26.2.4-1.el8 rabbitmq_erlang 21 M 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 21 M 2024-04-20T15:56:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 37 M 2024-04-20T15:56:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:56:15.845 INFO:teuthology.orchestra.run.smithi071.stdout:erlang-26.2.4-1.el8.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-04-20T15:56:15.847 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:56:15.847 INFO:teuthology.orchestra.run.smithi071.stdout:Total 18 MB/s | 21 MB 00:01 2024-04-20T15:56:15.848 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:56:15.858 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:56:15.858 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:56:16.079 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:56:16.082 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:56:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:56:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : erlang-26.2.4-1.el8.x86_64 1/1 2024-04-20T15:56:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: erlang-26.2.4-1.el8.x86_64 1/1 2024-04-20T15:56:19.199 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : erlang-26.2.4-1.el8.x86_64 1/1 2024-04-20T15:56:19.199 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:19.199 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:56:19.200 INFO:teuthology.orchestra.run.smithi071.stdout: erlang-26.2.4-1.el8.x86_64 2024-04-20T15:56:19.200 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:19.200 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:56:19.335 DEBUG:teuthology.orchestra.run.smithi071:> curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash 2024-04-20T15:56:19.684 INFO:teuthology.orchestra.run.smithi071.stdout:Detected operating system as centos/8. 2024-04-20T15:56:19.684 INFO:teuthology.orchestra.run.smithi071.stdout:Checking for curl... 2024-04-20T15:56:19.684 INFO:teuthology.orchestra.run.smithi071.stdout:Detected curl... 2024-04-20T15:56:19.684 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading repository file: https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/config_file.repo?os=centos&dist=8&source=script 2024-04-20T15:56:20.016 INFO:teuthology.orchestra.run.smithi071.stdout:done. 2024-04-20T15:56:20.017 INFO:teuthology.orchestra.run.smithi071.stdout:Installing yum-utils... 2024-04-20T15:56:22.134 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq_rabbitmq-server-source 605 B/s | 819 B 00:01 2024-04-20T15:56:22.544 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq_rabbitmq-server-source 9.4 kB/s | 3.9 kB 00:00 2024-04-20T15:56:22.573 INFO:teuthology.orchestra.run.smithi071.stderr:Importing GPG key 0x4D206F89: 2024-04-20T15:56:22.573 INFO:teuthology.orchestra.run.smithi071.stderr: Userid : "https://packagecloud.io/rabbitmq/rabbitmq-server (https://packagecloud.io/docs#gpg_signing) " 2024-04-20T15:56:22.573 INFO:teuthology.orchestra.run.smithi071.stderr: Fingerprint: 8C69 5B02 19AF DEB0 4A05 8ED8 F4E7 8920 4D20 6F89 2024-04-20T15:56:22.573 INFO:teuthology.orchestra.run.smithi071.stderr: From : https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey 2024-04-20T15:56:24.682 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq_rabbitmq-server-source 141 B/s | 296 B 00:02 2024-04-20T15:56:26.742 INFO:teuthology.orchestra.run.smithi071.stdout:Package yum-utils-4.0.21-25.el8.noarch is already installed. 2024-04-20T15:56:26.792 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:56:26.793 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-04-20T15:56:26.793 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:56:27.489 INFO:teuthology.orchestra.run.smithi071.stdout:Generating yum cache for rabbitmq_rabbitmq-server... 2024-04-20T15:56:29.416 INFO:teuthology.orchestra.run.smithi071.stderr:Importing GPG key 0x4D206F89: 2024-04-20T15:56:29.416 INFO:teuthology.orchestra.run.smithi071.stderr: Userid : "https://packagecloud.io/rabbitmq/rabbitmq-server (https://packagecloud.io/docs#gpg_signing) " 2024-04-20T15:56:29.417 INFO:teuthology.orchestra.run.smithi071.stderr: Fingerprint: 8C69 5B02 19AF DEB0 4A05 8ED8 F4E7 8920 4D20 6F89 2024-04-20T15:56:29.417 INFO:teuthology.orchestra.run.smithi071.stderr: From : https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey 2024-04-20T15:56:32.405 INFO:teuthology.orchestra.run.smithi071.stdout:Generating yum cache for rabbitmq_rabbitmq-server-source... 2024-04-20T15:56:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:33.938 INFO:teuthology.orchestra.run.smithi071.stdout:The repository is setup! You can now install packages. 2024-04-20T15:56:33.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rabbitmq-server 2024-04-20T15:56:34.773 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:01 ago on Sat 20 Apr 2024 03:56:33 PM UTC. 2024-04-20T15:56:36.514 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:56:36.514 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:56:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: rabbitmq-server noarch 3.13.1-1.el8 rabbitmq_rabbitmq-server 18 M 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-04-20T15:56:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:36.516 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 18 M 2024-04-20T15:56:36.516 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 23 M 2024-04-20T15:56:36.516 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:56:37.688 INFO:teuthology.orchestra.run.smithi071.stdout:rabbitmq-server-3.13.1-1.el8.noarch.rpm 15 MB/s | 18 MB 00:01 2024-04-20T15:56:37.690 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:56:37.690 INFO:teuthology.orchestra.run.smithi071.stdout:Total 15 MB/s | 18 MB 00:01 2024-04-20T15:56:37.691 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:56:37.705 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:56:37.705 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:56:37.803 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:56:37.806 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:56:37.957 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:56:39.317 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T15:56:39.568 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T15:56:40.023 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T15:56:40.023 INFO:teuthology.orchestra.run.smithi071.stdout:[/usr/lib/tmpfiles.d/rabbitmq-server.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rabbitmq → /run/rabbitmq; please update the tmpfiles.d/ drop-in file accordingly. 2024-04-20T15:56:40.023 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:40.652 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T15:56:40.652 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:40.652 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:56:40.653 INFO:teuthology.orchestra.run.smithi071.stdout: rabbitmq-server-3.13.1-1.el8.noarch 2024-04-20T15:56:40.653 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:56:40.653 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:56:40.851 INFO:tasks.rabbitmq:Bringing up Daemon and RabbitMQ service... 2024-04-20T15:56:40.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo chkconfig rabbitmq-server on 2024-04-20T15:56:40.900 INFO:teuthology.orchestra.run.smithi071.stderr:Note: Forwarding request to 'systemctl enable rabbitmq-server.service'. 2024-04-20T15:56:40.905 INFO:teuthology.orchestra.run.smithi071.stderr:Created symlink /etc/systemd/system/multi-user.target.wants/rabbitmq-server.service → /usr/lib/systemd/system/rabbitmq-server.service. 2024-04-20T15:56:40.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/service rabbitmq-server start 2024-04-20T15:56:41.036 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl start rabbitmq-server.service 2024-04-20T15:56:44.840 INFO:teuthology.run_tasks:Running task notification-tests... 2024-04-20T15:56:44.854 DEBUG:tasks.notification_tests:Notifications config is {'client.0': {'extra_attr': ['amqp_test'], 'rgw_server': 'client.0'}} 2024-04-20T15:56:44.854 INFO:tasks.notification_tests:Downloading bucket-notifications-tests... 2024-04-20T15:56:44.854 INFO:tasks.notification_tests:Using branch wip-yuri4-testing-2024-04-19-0708-quincy from https://github.com/ceph/ceph-ci.git for bucket notifications tests 2024-04-20T15:56:44.854 DEBUG:teuthology.orchestra.run.smithi071:> git clone -b wip-yuri4-testing-2024-04-19-0708-quincy https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/ceph 2024-04-20T15:56:44.899 INFO:teuthology.orchestra.run.smithi071.stderr:Cloning into '/home/ubuntu/cephtest/ceph'... 2024-04-20T15:57:37.119 INFO:tasks.notification_tests:Pre-processing... 2024-04-20T15:57:37.119 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p /home/ubuntu/.aws/models/s3/2006-03-01/ 2024-04-20T15:57:37.179 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/.aws/models/s3/2006-03-01/ && cp /home/ubuntu/cephtest/ceph/examples/boto3/service-2.sdk-extras.json service-2.sdk-extras.json 2024-04-20T15:57:37.238 INFO:tasks.notification_tests:Creating rgw user... 2024-04-20T15:57:37.239 DEBUG:tasks.notification_tests:Creating user foo.client.0 on client.0 2024-04-20T15:57:37.239 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key RPTKEXNTSFXEXSLLARJY --secret Bi1Jn0rvZOTYkgmL+Qty+OS6cgKOfCCSzHkGoY/AvMKdxxB1/dTYrQ== --cluster ceph 2024-04-20T15:57:37.330 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T15:57:37.330 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T15:57:37.332 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.332+0000 7f9727b46980 -1 asok(0x56308741b870) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.0.53668.asok': (13) Permission denied 2024-04-20T15:57:37.336 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.336+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 realm 2024-04-20T15:57:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-20T15:57:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.338+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.339+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:57:37.339 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.339+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.340+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=1004 2024-04-20T15:57:37.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.340+0000 7f9727b46980 20 searching for the correct realm 2024-04-20T15:57:37.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:57:37.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:57:37.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:57:37.351 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:57:37.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.352 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.351+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=396 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.352+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.353 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.353+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:57:37.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 RGWRados::pool_iterate: got zonegroup_info.3221e4dc-597a-4adc-b880-a5fd67f97b2e 2024-04-20T15:57:37.361 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 RGWRados::pool_iterate: got zone_info.f76a52bc-49fc-4b9b-bb36-9008f7b8b677 2024-04-20T15:57:37.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-20T15:57:37.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-20T15:57:37.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:57:37.362 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.361+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.362+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=1004 2024-04-20T15:57:37.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.362+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.362+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-20T15:57:37.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.362+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.362+0000 7f9727b46980 20 rados_obj.operate() r=0 bl.length=396 2024-04-20T15:57:37.363 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.362+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 zone default found 2024-04-20T15:57:37.364 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 4 Realm: () 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 4 ZoneGroup: default (3221e4dc-597a-4adc-b880-a5fd67f97b2e) 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 4 Zone: default (f76a52bc-49fc-4b9b-bb36-9008f7b8b677) 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 zonegroup default 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.363+0000 7f9727b46980 20 rados->read ofs=0 len=0 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.364+0000 7f9727b46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.364+0000 7f9727b46980 20 started sync module instance, tier type = 2024-04-20T15:57:37.365 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.364+0000 7f9727b46980 20 started zone id=f76a52bc-49fc-4b9b-bb36-9008f7b8b677 (name=default) with tier type = 2024-04-20T15:57:37.372 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=0 2024-04-20T15:57:37.372 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=1 2024-04-20T15:57:37.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=2 2024-04-20T15:57:37.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=3 2024-04-20T15:57:37.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=4 2024-04-20T15:57:37.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=5 2024-04-20T15:57:37.373 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.372+0000 7f9727b46980 20 add_watcher() i=6 2024-04-20T15:57:37.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.374+0000 7f9727b46980 20 add_watcher() i=7 2024-04-20T15:57:37.374 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.374+0000 7f9727b46980 2 all 8 watchers are set, enabling cache 2024-04-20T15:57:37.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.376+0000 7f96f0fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-20T15:57:37.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.376+0000 7f9727b46980 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-20T15:57:37.376 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.376+0000 7f9727b46980 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-20T15:57:37.381 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.381+0000 7f9727b46980 5 note: GC not initialized 2024-04-20T15:57:37.382 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f965f7fe700 20 reqs_thread_entry: start 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f965d7fa700 20 reqs_thread_entry: start 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f9727b46980 20 init_complete bucket index max shards: 11 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f9727b46980 10 rgw notify: Started notification manager with: 1 workers 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f964b7fe700 20 rgw notify: INFO: next queues processing will happen at: Sat Apr 20 15:58:07 2024 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f964affd700 20 reqs_thread_entry: start 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f9727b46980 20 get_system_obj_state: rctx=0x7ffd94e1a818 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5630873ea240 s->prefetch_data=0 2024-04-20T15:57:37.383 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.382+0000 7f9727b46980 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 20 get_system_obj_state: rctx=0x7ffd94e1a528 obj=default.rgw.meta:users.keys:RPTKEXNTSFXEXSLLARJY state=0x5630873ea240 s->prefetch_data=0 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 cache get: name=default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY : miss 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 cache put: name=default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY info.flags=0x0 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 adding default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY to cache LRU end 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 20 get_system_obj_state: rctx=0x7ffd94e19dd8 obj=default.rgw.meta:users.keys:RPTKEXNTSFXEXSLLARJY state=0x5630873ea240 s->prefetch_data=0 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 cache get: name=default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY : hit (negative entry) 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 20 get_system_obj_state: rctx=0x7ffd94e1a468 obj=default.rgw.meta:users.keys:RPTKEXNTSFXEXSLLARJY state=0x5630873ea240 s->prefetch_data=0 2024-04-20T15:57:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.383+0000 7f9727b46980 10 cache get: name=default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY : hit (negative entry) 2024-04-20T15:57:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.384+0000 7f9727b46980 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-04-20T15:57:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.384+0000 7f9727b46980 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-20T15:57:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.384+0000 7f9727b46980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-04-20T15:57:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.385+0000 7f9719095700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822656 cookie 94766429111952 notifier 4457 bl.length()=594 2024-04-20T15:57:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.385+0000 7f9719095700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-04-20T15:57:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.385+0000 7f9719095700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-04-20T15:57:37.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.386+0000 7f9727b46980 10 cache put: name=default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY info.flags=0x7 2024-04-20T15:57:37.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.386+0000 7f9727b46980 10 moving default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY to cache LRU end 2024-04-20T15:57:37.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.386+0000 7f9727b46980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:RPTKEXNTSFXEXSLLARJY, ofs0, ns] 2024-04-20T15:57:37.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.387+0000 7f9719095700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822657 cookie 94766429131296 notifier 4457 bl.length()=186 2024-04-20T15:57:37.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.387+0000 7f9719095700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY info.flags=0x7 2024-04-20T15:57:37.387 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.387+0000 7f9719095700 10 rgw watcher librados: moving default.rgw.meta+users.keys+RPTKEXNTSFXEXSLLARJY to cache LRU end 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "user_id": "foo.client.0", 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "display_name": "Mr. foo.client.0", 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "email": "", 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "suspended": 0, 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "max_buckets": 1000, 2024-04-20T15:57:37.388 INFO:teuthology.orchestra.run.smithi071.stdout: "subusers": [], 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "keys": [ 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "user": "foo.client.0", 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "access_key": "RPTKEXNTSFXEXSLLARJY", 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "secret_key": "Bi1Jn0rvZOTYkgmL+Qty+OS6cgKOfCCSzHkGoY/AvMKdxxB1/dTYrQ==" 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "swift_keys": [], 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": [], 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "op_mask": "read, write, delete", 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "default_placement": "", 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "default_storage_class": "", 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "placement_tags": [], 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "bucket_quota": { 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2024-04-20T15:57:37.389 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "user_quota": { 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "enabled": false, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "check_on_raw": false, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size": -1, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "max_size_kb": 0, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "max_objects": -1 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "temp_url_keys": [], 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "rgw", 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout: "mfa_ids": [] 2024-04-20T15:57:37.390 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-20T15:57:37.391 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:57:37.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.389+0000 7f9727b46980 20 remove_watcher() i=0 2024-04-20T15:57:37.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.389+0000 7f9727b46980 2 removed watcher, disabling cache 2024-04-20T15:57:37.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.390+0000 7f9727b46980 20 remove_watcher() i=1 2024-04-20T15:57:37.391 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.391+0000 7f9727b46980 20 remove_watcher() i=2 2024-04-20T15:57:37.392 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.392+0000 7f9727b46980 20 remove_watcher() i=3 2024-04-20T15:57:37.393 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.393+0000 7f9727b46980 20 remove_watcher() i=4 2024-04-20T15:57:37.394 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.394+0000 7f9727b46980 20 remove_watcher() i=5 2024-04-20T15:57:37.395 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.395+0000 7f9727b46980 20 remove_watcher() i=6 2024-04-20T15:57:37.396 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.396+0000 7f9727b46980 20 remove_watcher() i=7 2024-04-20T15:57:37.396 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-20T15:57:37.396+0000 7f96f0fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-20T15:57:37.402 INFO:tasks.notification_tests:Configuring bucket-notifications-tests... 2024-04-20T15:57:37.402 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-20T15:57:37.402 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf 2024-04-20T15:57:37.420 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification && ./bootstrap 2024-04-20T15:57:46.635 INFO:teuthology.orchestra.run.smithi071.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2024-04-20T15:57:47.430 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:14 ago on Sat 20 Apr 2024 03:56:33 PM UTC. 2024-04-20T15:57:49.017 INFO:teuthology.orchestra.run.smithi071.stdout:Package python36-devel-3.6.8-38.module_el8.5.0+895+a459eca8.x86_64 is already installed. 2024-04-20T15:57:49.082 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T15:57:49.085 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:57:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-04-20T15:57:49.085 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:57:49.085 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-20T15:57:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2024-04-20T15:57:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: libxml2-devel x86_64 2.9.7-18.el8 appstream 1.1 M 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pika noarch 1.0.1-3.el8.1 epel 258 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: python3-xmltodict noarch 0.12.0-6.el8 epel 24 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: libxml2 x86_64 2.9.7-18.el8 baseos 697 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: python3-libxml2 x86_64 2.9.7-18.el8 baseos 238 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: python36-devel x86_64 3.6.8-39.module_el8+762+77bd8591 appstream 17 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: cmake-filesystem x86_64 3.26.5-2.el8 appstream 52 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2024-04-20T15:57:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-Automat noarch 0.7.0-3.1.el8 epel 66 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-attrs noarch 17.4.0-6.el8 appstream 53 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-constantly noarch 15.1.0-4.el8 epel 19 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-hamcrest noarch 1.9.0-8.el8 epel 75 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-hyperlink noarch 19.0.0-1.el8.1 epel 75 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-incremental noarch 17.5.0-5.el8.1 epel 33 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycurl x86_64 7.43.0.2-4.el8 appstream 227 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zope-event noarch 4.2.0-12.el8 epel 210 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zope-interface x86_64 4.6.0-1.el8 epel 158 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-service-identity 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: noarch 18.1.0-2.el8 epel 28 k 2024-04-20T15:57:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tornado x86_64 6.0.2-1.el8 epel 722 k 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: python3-twisted x86_64 19.10.0-4.el8 epel 6.8 M 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout:Install 22 Packages 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 4 Packages 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:57:49.088 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 12 M 2024-04-20T15:57:49.089 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-20T15:57:49.613 INFO:teuthology.orchestra.run.smithi071.stdout:(1/26): cmake-filesystem-3.26.5-2.el8.x86_64.rp 278 kB/s | 52 kB 00:00 2024-04-20T15:57:49.638 INFO:teuthology.orchestra.run.smithi071.stdout:(2/26): libevent-devel-2.1.8-5.el8.x86_64.rpm 491 kB/s | 104 kB 00:00 2024-04-20T15:57:49.730 INFO:teuthology.orchestra.run.smithi071.stdout:(3/26): python3-attrs-17.4.0-6.el8.noarch.rpm 577 kB/s | 53 kB 00:00 2024-04-20T15:57:49.999 INFO:teuthology.orchestra.run.smithi071.stdout:(4/26): libxslt-devel-1.1.32-6.el8.x86_64.rpm 836 kB/s | 323 kB 00:00 2024-04-20T15:57:50.030 INFO:teuthology.orchestra.run.smithi071.stdout:(5/26): python3-pycurl-7.43.0.2-4.el8.x86_64.rp 758 kB/s | 227 kB 00:00 2024-04-20T15:57:50.064 INFO:teuthology.orchestra.run.smithi071.stdout:(6/26): libxml2-devel-2.9.7-18.el8.x86_64.rpm 1.7 MB/s | 1.1 MB 00:00 2024-04-20T15:57:50.239 INFO:teuthology.orchestra.run.smithi071.stdout:(7/26): libgpg-error-devel-1.31-1.el8.x86_64.rp 306 kB/s | 64 kB 00:00 2024-04-20T15:57:50.266 INFO:teuthology.orchestra.run.smithi071.stdout:(8/26): xz-devel-5.2.4-4.el8.x86_64.rpm 312 kB/s | 63 kB 00:00 2024-04-20T15:57:50.290 INFO:teuthology.orchestra.run.smithi071.stdout:(9/26): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 515 kB/s | 150 kB 00:00 2024-04-20T15:57:50.307 INFO:teuthology.orchestra.run.smithi071.stdout:(10/26): zlib-devel-1.2.11-25.el8.x86_64.rpm 871 kB/s | 59 kB 00:00 2024-04-20T15:57:50.424 INFO:teuthology.orchestra.run.smithi071.stdout:(11/26): python3-constantly-15.1.0-4.el8.noarch 144 kB/s | 19 kB 00:00 2024-04-20T15:57:50.449 INFO:teuthology.orchestra.run.smithi071.stdout:(12/26): python3-Automat-0.7.0-3.1.el8.noarch.r 358 kB/s | 66 kB 00:00 2024-04-20T15:57:50.475 INFO:teuthology.orchestra.run.smithi071.stdout:(13/26): python3-hamcrest-1.9.0-8.el8.noarch.rp 447 kB/s | 75 kB 00:00 2024-04-20T15:57:50.508 INFO:teuthology.orchestra.run.smithi071.stdout:(14/26): python3-hyperlink-19.0.0-1.el8.1.noarc 895 kB/s | 75 kB 00:00 2024-04-20T15:57:50.534 INFO:teuthology.orchestra.run.smithi071.stdout:(15/26): python3-incremental-17.5.0-5.el8.1.noa 387 kB/s | 33 kB 00:00 2024-04-20T15:57:50.592 INFO:teuthology.orchestra.run.smithi071.stdout:(16/26): python3-service-identity-18.1.0-2.el8. 336 kB/s | 28 kB 00:00 2024-04-20T15:57:50.660 INFO:teuthology.orchestra.run.smithi071.stdout:(17/26): python3-pika-1.0.1-3.el8.1.noarch.rpm 1.4 MB/s | 258 kB 00:00 2024-04-20T15:57:50.710 INFO:teuthology.orchestra.run.smithi071.stdout:(18/26): python3-xmltodict-0.12.0-6.el8.noarch. 490 kB/s | 24 kB 00:00 2024-04-20T15:57:50.794 INFO:teuthology.orchestra.run.smithi071.stdout:(19/26): python3-zope-event-4.2.0-12.el8.noarch 2.5 MB/s | 210 kB 00:00 2024-04-20T15:57:50.819 INFO:teuthology.orchestra.run.smithi071.stdout:(20/26): python3-tornado-6.0.2-1.el8.x86_64.rpm 2.5 MB/s | 722 kB 00:00 2024-04-20T15:57:50.877 INFO:teuthology.orchestra.run.smithi071.stdout:(21/26): python36-devel-3.6.8-39.module_el8+762 284 kB/s | 17 kB 00:00 2024-04-20T15:57:50.902 INFO:teuthology.orchestra.run.smithi071.stdout:(22/26): python3-zope-interface-4.6.0-1.el8.x86 1.4 MB/s | 158 kB 00:00 2024-04-20T15:57:51.061 INFO:teuthology.orchestra.run.smithi071.stdout:(23/26): python3-libxml2-2.9.7-18.el8.x86_64.rp 1.5 MB/s | 238 kB 00:00 2024-04-20T15:57:51.095 INFO:teuthology.orchestra.run.smithi071.stdout:(24/26): libxml2-2.9.7-18.el8.x86_64.rpm 3.1 MB/s | 697 kB 00:00 2024-04-20T15:57:51.120 INFO:teuthology.orchestra.run.smithi071.stdout:(25/26): zlib-1.2.11-25.el8.x86_64.rpm 1.7 MB/s | 103 kB 00:00 2024-04-20T15:57:51.896 INFO:teuthology.orchestra.run.smithi071.stdout:(26/26): python3-twisted-19.10.0-4.el8.x86_64.r 5.2 MB/s | 6.8 MB 00:01 2024-04-20T15:57:51.896 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-20T15:57:51.896 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.1 MB/s | 12 MB 00:02 2024-04-20T15:57:52.025 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T15:57:52.177 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T15:57:52.177 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T15:57:52.439 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T15:57:52.442 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T15:57:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T15:57:53.617 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 1/30 2024-04-20T15:57:53.875 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-attrs-17.4.0-6.el8.noarch 2/30 2024-04-20T15:57:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libxml2-2.9.7-18.el8.x86_64 3/30 2024-04-20T15:57:54.268 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 4/30 2024-04-20T15:57:54.302 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 4/30 2024-04-20T15:57:54.518 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 5/30 2024-04-20T15:57:54.545 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 5/30 2024-04-20T15:57:54.796 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-Automat-0.7.0-3.1.el8.noarch 6/30 2024-04-20T15:57:55.084 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-service-identity-18.1.0-2.el8.noarch 7/30 2024-04-20T15:57:55.315 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 8/30 2024-04-20T15:57:55.592 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zope-event-4.2.0-12.el8.noarch 9/30 2024-04-20T15:57:55.849 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zope-interface-4.6.0-1.el8.x86_64 10/30 2024-04-20T15:57:56.107 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-incremental-17.5.0-5.el8.1.noarch 11/30 2024-04-20T15:57:56.415 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-hyperlink-19.0.0-1.el8.1.noarch 12/30 2024-04-20T15:57:56.693 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-hamcrest-1.9.0-8.el8.noarch 13/30 2024-04-20T15:57:58.539 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-constantly-15.1.0-4.el8.noarch 14/30 2024-04-20T15:57:58.898 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-twisted-19.10.0-4.el8.x86_64 15/30 2024-04-20T15:57:59.194 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 16/30 2024-04-20T15:57:59.639 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycurl-7.43.0.2-4.el8.x86_64 17/30 2024-04-20T15:57:59.957 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tornado-6.0.2-1.el8.x86_64 18/30 2024-04-20T15:58:00.480 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cmake-filesystem-3.26.5-2.el8.x86_64 19/30 2024-04-20T15:58:00.782 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libxml2-devel-2.9.7-18.el8.x86_64 20/30 2024-04-20T15:58:01.066 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 21/30 2024-04-20T15:58:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pika-1.0.1-3.el8.1.noarch 22/30 2024-04-20T15:58:01.646 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : python3-libxml2-2.9.7-18.el8.x86_64 23/30 2024-04-20T15:58:01.875 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : python36-devel-3.6.8-39.module_el8+762+77bd8591.x8 24/30 2024-04-20T15:58:01.888 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python36-devel-3.6.8-39.module_el8+762+77bd8591.x8 24/30 2024-04-20T15:58:02.131 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-xmltodict-0.12.0-6.el8.noarch 25/30 2024-04-20T15:58:02.326 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 26/30 2024-04-20T15:58:02.326 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : python36-devel-3.6.8-38.module_el8.5.0+895+a459eca 27/30 2024-04-20T15:58:02.527 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: python36-devel-3.6.8-38.module_el8.5.0+895+a459eca 27/30 2024-04-20T15:58:02.718 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : python3-libxml2-2.9.7-16.el8.x86_64 28/30 2024-04-20T15:58:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libxml2-2.9.7-16.el8.x86_64 29/30 2024-04-20T15:58:03.092 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 30/30 2024-04-20T15:58:03.459 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 30/30 2024-04-20T15:58:03.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cmake-filesystem-3.26.5-2.el8.x86_64 1/30 2024-04-20T15:58:03.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/30 2024-04-20T15:58:03.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libxml2-devel-2.9.7-18.el8.x86_64 3/30 2024-04-20T15:58:03.459 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-attrs-17.4.0-6.el8.noarch 5/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycurl-7.43.0.2-4.el8.x86_64 6/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 7/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 8/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 9/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 10/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-Automat-0.7.0-3.1.el8.noarch 11/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-constantly-15.1.0-4.el8.noarch 12/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-hamcrest-1.9.0-8.el8.noarch 13/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-hyperlink-19.0.0-1.el8.1.noarch 14/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-incremental-17.5.0-5.el8.1.noarch 15/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pika-1.0.1-3.el8.1.noarch 16/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-service-identity-18.1.0-2.el8.noarch 17/30 2024-04-20T15:58:03.460 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tornado-6.0.2-1.el8.x86_64 18/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-twisted-19.10.0-4.el8.x86_64 19/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-xmltodict-0.12.0-6.el8.noarch 20/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zope-event-4.2.0-12.el8.noarch 21/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zope-interface-4.6.0-1.el8.x86_64 22/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python36-devel-3.6.8-39.module_el8+762+77bd8591.x8 23/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python36-devel-3.6.8-38.module_el8.5.0+895+a459eca 24/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libxml2-2.9.7-18.el8.x86_64 25/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libxml2-2.9.7-16.el8.x86_64 26/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-libxml2-2.9.7-18.el8.x86_64 27/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-libxml2-2.9.7-16.el8.x86_64 28/30 2024-04-20T15:58:03.461 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 29/30 2024-04-20T15:58:04.175 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 30/30 2024-04-20T15:58:04.175 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:04.175 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-04-20T15:58:04.175 INFO:teuthology.orchestra.run.smithi071.stdout: libxml2-2.9.7-18.el8.x86_64 2024-04-20T15:58:04.175 INFO:teuthology.orchestra.run.smithi071.stdout: python3-libxml2-2.9.7-18.el8.x86_64 2024-04-20T15:58:04.175 INFO:teuthology.orchestra.run.smithi071.stdout: python36-devel-3.6.8-39.module_el8+762+77bd8591.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-1.2.11-25.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: cmake-filesystem-3.26.5-2.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: libevent-devel-2.1.8-5.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: libgcrypt-devel-1.8.5-7.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: libgpg-error-devel-1.31-1.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: libxml2-devel-2.9.7-18.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: libxslt-devel-1.1.32-6.el8.x86_64 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-Automat-0.7.0-3.1.el8.noarch 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-attrs-17.4.0-6.el8.noarch 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-constantly-15.1.0-4.el8.noarch 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-hamcrest-1.9.0-8.el8.noarch 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-hyperlink-19.0.0-1.el8.1.noarch 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-incremental-17.5.0-5.el8.1.noarch 2024-04-20T15:58:04.176 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pika-1.0.1-3.el8.1.noarch 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycurl-7.43.0.2-4.el8.x86_64 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-service-identity-18.1.0-2.el8.noarch 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tornado-6.0.2-1.el8.x86_64 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-twisted-19.10.0-4.el8.x86_64 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-xmltodict-0.12.0-6.el8.noarch 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zope-event-4.2.0-12.el8.noarch 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zope-interface-4.6.0-1.el8.x86_64 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: xz-devel-5.2.4-4.el8.x86_64 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel-1.2.11-25.el8.x86_64 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:04.177 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T15:58:06.560 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting pip 2024-04-20T15:58:06.692 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB) 2024-04-20T15:58:06.933 INFO:teuthology.orchestra.run.smithi071.stdout:Installing collected packages: pip 2024-04-20T15:58:06.934 INFO:teuthology.orchestra.run.smithi071.stdout: Found existing installation: pip 9.0.3 2024-04-20T15:58:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: Uninstalling pip-9.0.3: 2024-04-20T15:58:07.126 INFO:teuthology.orchestra.run.smithi071.stdout: Successfully uninstalled pip-9.0.3 2024-04-20T15:58:07.765 INFO:teuthology.orchestra.run.smithi071.stdout:Successfully installed pip-21.3.1 2024-04-20T15:58:08.015 INFO:teuthology.orchestra.run.smithi071.stderr:You are using pip version 21.3.1, however version 24.0 is available. 2024-04-20T15:58:08.015 INFO:teuthology.orchestra.run.smithi071.stderr:You should consider upgrading via the 'pip install --upgrade pip' command. 2024-04-20T15:58:08.426 INFO:teuthology.orchestra.run.smithi071.stdout:Requirement already satisfied: nose>=1.0.0 in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.3.7) 2024-04-20T15:58:08.629 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting boto>=2.6.0 2024-04-20T15:58:08.690 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2024-04-20T15:58:09.899 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting boto3>=1.0.0 2024-04-20T15:58:09.920 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2024-04-20T15:58:10.020 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting configparser>=5.0.0 2024-04-20T15:58:10.038 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading configparser-5.2.0-py3-none-any.whl (19 kB) 2024-04-20T15:58:10.163 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting kafka-python>=2.0.0 2024-04-20T15:58:10.181 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading kafka_python-2.0.2-py2.py3-none-any.whl (246 kB) 2024-04-20T15:58:10.209 INFO:teuthology.orchestra.run.smithi071.stdout:Requirement already satisfied: pika in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (1.0.1) 2024-04-20T15:58:10.405 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting pika 2024-04-20T15:58:10.423 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading pika-1.3.1-py3-none-any.whl (155 kB) 2024-04-20T15:58:10.808 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting cloudevents 2024-04-20T15:58:10.834 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading cloudevents-1.10.1-py3-none-any.whl (55 kB) 2024-04-20T15:58:11.065 INFO:teuthology.orchestra.run.smithi071.stdout:Requirement already satisfied: xmltodict in /usr/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (0.12.0) 2024-04-20T15:58:11.244 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting xmltodict 2024-04-20T15:58:11.261 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB) 2024-04-20T15:58:11.740 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting jmespath<2.0.0,>=0.7.1 2024-04-20T15:58:11.759 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-04-20T15:58:13.043 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting botocore<1.27.0,>=1.26.10 2024-04-20T15:58:13.073 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2024-04-20T15:58:13.653 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2024-04-20T15:58:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-04-20T15:58:13.807 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting deprecation<3.0,>=2.0 2024-04-20T15:58:13.826 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB) 2024-04-20T15:58:13.858 INFO:teuthology.orchestra.run.smithi071.stdout:Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.6/site-packages (from botocore<1.27.0,>=1.26.10->boto3>=1.0.0->-r requirements.txt (line 3)) (2.6.1) 2024-04-20T15:58:13.974 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting urllib3<1.27,>=1.25.4 2024-04-20T15:58:13.993 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-04-20T15:58:14.100 INFO:teuthology.orchestra.run.smithi071.stdout:Collecting packaging 2024-04-20T15:58:14.118 INFO:teuthology.orchestra.run.smithi071.stdout: Downloading packaging-21.3-py3-none-any.whl (40 kB) 2024-04-20T15:58:14.330 INFO:teuthology.orchestra.run.smithi071.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.27.0,>=1.26.10->boto3>=1.0.0->-r requirements.txt (line 3)) (1.11.0) 2024-04-20T15:58:14.351 INFO:teuthology.orchestra.run.smithi071.stdout:Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3.6/site-packages (from packaging->deprecation<3.0,>=2.0->cloudevents->-r requirements.txt (line 7)) (2.1.10) 2024-04-20T15:58:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Installing collected packages: urllib3, jmespath, packaging, botocore, s3transfer, deprecation, xmltodict, pika, kafka-python, configparser, cloudevents, boto3, boto 2024-04-20T15:58:14.574 INFO:teuthology.orchestra.run.smithi071.stdout: Attempting uninstall: urllib3 2024-04-20T15:58:14.574 INFO:teuthology.orchestra.run.smithi071.stdout: Found existing installation: urllib3 1.24.2 2024-04-20T15:58:14.575 INFO:teuthology.orchestra.run.smithi071.stdout: Not uninstalling urllib3 at /usr/lib/python3.6/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2024-04-20T15:58:14.575 INFO:teuthology.orchestra.run.smithi071.stdout: Can't uninstall 'urllib3'. No files were found to uninstall. 2024-04-20T15:58:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: Attempting uninstall: xmltodict 2024-04-20T15:58:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: Found existing installation: xmltodict 0.12.0 2024-04-20T15:58:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: Not uninstalling xmltodict at /usr/lib/python3.6/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2024-04-20T15:58:15.575 INFO:teuthology.orchestra.run.smithi071.stdout: Can't uninstall 'xmltodict'. No files were found to uninstall. 2024-04-20T15:58:15.625 INFO:teuthology.orchestra.run.smithi071.stdout: Attempting uninstall: pika 2024-04-20T15:58:15.626 INFO:teuthology.orchestra.run.smithi071.stdout: Found existing installation: pika 1.0.1 2024-04-20T15:58:15.626 INFO:teuthology.orchestra.run.smithi071.stdout: Not uninstalling pika at /usr/lib/python3.6/site-packages, outside environment /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv 2024-04-20T15:58:15.626 INFO:teuthology.orchestra.run.smithi071.stdout: Can't uninstall 'pika'. No files were found to uninstall. 2024-04-20T15:58:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-04-20T15:58:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:requests 2.20.0 requires urllib3<1.25,>=1.21.1, but you have urllib3 1.26.18 which is incompatible. 2024-04-20T15:58:16.615 INFO:teuthology.orchestra.run.smithi071.stdout:Successfully installed boto-2.49.0 boto3-1.23.10 botocore-1.26.10 cloudevents-1.10.1 configparser-5.2.0 deprecation-2.1.0 jmespath-0.10.0 kafka-python-2.0.2 packaging-21.3 pika-1.3.1 s3transfer-0.5.2 urllib3-1.26.18 xmltodict-0.13.0 2024-04-20T15:58:17.141 INFO:teuthology.orchestra.run.smithi071.stdout:running develop 2024-04-20T15:58:17.226 INFO:teuthology.orchestra.run.smithi071.stdout:running egg_info 2024-04-20T15:58:17.226 INFO:teuthology.orchestra.run.smithi071.stdout:creating bn_tests.egg-info 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:writing bn_tests.egg-info/PKG-INFO 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:writing dependency_links to bn_tests.egg-info/dependency_links.txt 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:writing requirements to bn_tests.egg-info/requires.txt 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:writing top-level names to bn_tests.egg-info/top_level.txt 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:writing manifest file 'bn_tests.egg-info/SOURCES.txt' 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:reading manifest file 'bn_tests.egg-info/SOURCES.txt' 2024-04-20T15:58:17.227 INFO:teuthology.orchestra.run.smithi071.stdout:writing manifest file 'bn_tests.egg-info/SOURCES.txt' 2024-04-20T15:58:17.234 INFO:teuthology.orchestra.run.smithi071.stdout:running build_ext 2024-04-20T15:58:17.243 INFO:teuthology.orchestra.run.smithi071.stdout:Creating /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages/bn-tests.egg-link (link to .) 2024-04-20T15:58:17.244 INFO:teuthology.orchestra.run.smithi071.stdout:Adding bn-tests 0.0.1 to easy-install.pth file 2024-04-20T15:58:17.244 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.244 INFO:teuthology.orchestra.run.smithi071.stdout:Installed /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification 2024-04-20T15:58:17.245 INFO:teuthology.orchestra.run.smithi071.stdout:Processing dependencies for bn-tests==0.0.1 2024-04-20T15:58:17.269 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for boto3==1.23.10 2024-04-20T15:58:17.269 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: boto3 1.23.10 2024-04-20T15:58:17.269 INFO:teuthology.orchestra.run.smithi071.stdout:Adding boto3 1.23.10 to easy-install.pth file 2024-04-20T15:58:17.269 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.269 INFO:teuthology.orchestra.run.smithi071.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages 2024-04-20T15:58:17.270 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for boto==2.49.0 2024-04-20T15:58:17.270 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: boto 2.49.0 2024-04-20T15:58:17.270 INFO:teuthology.orchestra.run.smithi071.stdout:Adding boto 2.49.0 to easy-install.pth file 2024-04-20T15:58:17.270 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.271 INFO:teuthology.orchestra.run.smithi071.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages 2024-04-20T15:58:17.271 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for s3transfer==0.5.2 2024-04-20T15:58:17.271 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: s3transfer 0.5.2 2024-04-20T15:58:17.271 INFO:teuthology.orchestra.run.smithi071.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2024-04-20T15:58:17.271 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.271 INFO:teuthology.orchestra.run.smithi071.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for jmespath==0.10.0 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: jmespath 0.10.0 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for botocore==1.26.10 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: botocore 1.26.10 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout:Adding botocore 1.26.10 to easy-install.pth file 2024-04-20T15:58:17.272 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.273 INFO:teuthology.orchestra.run.smithi071.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for urllib3==1.26.18 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: urllib3 1.26.18 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Adding urllib3 1.26.18 to easy-install.pth file 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Using /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib/python3.6/site-packages 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for python-dateutil==2.6.1 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: python-dateutil 2.6.1 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Adding python-dateutil 2.6.1 to easy-install.pth file 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.274 INFO:teuthology.orchestra.run.smithi071.stdout:Using /usr/lib/python3.6/site-packages 2024-04-20T15:58:17.275 INFO:teuthology.orchestra.run.smithi071.stdout:Searching for six==1.11.0 2024-04-20T15:58:17.275 INFO:teuthology.orchestra.run.smithi071.stdout:Best match: six 1.11.0 2024-04-20T15:58:17.275 INFO:teuthology.orchestra.run.smithi071.stdout:Adding six 1.11.0 to easy-install.pth file 2024-04-20T15:58:17.275 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T15:58:17.276 INFO:teuthology.orchestra.run.smithi071.stdout:Using /usr/lib/python3.6/site-packages 2024-04-20T15:58:17.276 INFO:teuthology.orchestra.run.smithi071.stdout:Finished processing dependencies for bn-tests==0.0.1 2024-04-20T15:58:17.294 INFO:tasks.notification_tests:Running bucket-notifications-tests... 2024-04-20T15:58:17.294 DEBUG:teuthology.orchestra.run.smithi071:bucket notification tests against different endpoints> BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test 2024-04-20T15:58:27.160 INFO:teuthology.orchestra.run.smithi071.stderr:test s3 notification filter on master ... /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2024-04-20T15:58:27.160 INFO:teuthology.orchestra.run.smithi071.stderr: warnings.warn(warning, PythonDeprecationWarning) 2024-04-20T15:58:27.160 INFO:teuthology.orchestra.run.smithi071.stderr:ok 2024-04-20T15:58:29.356 INFO:tasks.rgw.client.0.smithi071.stdout:*** Caught signal (Segmentation fault) ** 2024-04-20T15:58:29.356 INFO:tasks.rgw.client.0.smithi071.stdout: in thread 7f5fc4057700 thread_name:radosgw 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: ceph version 17.2.7-902-g7f4fa80c (7f4fa80c8c97f53f52b1189d2347b5761f6287e3) quincy (stable) 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f60dfddacf0] 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 2: /lib64/libradosgw.so.2(+0x90c5b1) [0x7f60e289e5b1] 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 3: (RGWObjManifest::~RGWObjManifest()+0x1d3) [0x7f60e28b5233] 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 4: (RGWObjState::~RGWObjState()+0x1e3) [0x7f60e2acf883] 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 5: /lib64/libradosgw.so.2(+0x6e46ee) [0x7f60e26766ee] 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 6: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, std::shared_ptr, int*)+0x1702) [0x7f60e267a882] 2024-04-20T15:58:29.359 INFO:tasks.rgw.client.0.smithi071.stdout: 7: /lib64/libradosgw.so.2(+0x645765) [0x7f60e25d7765] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 8: /lib64/libradosgw.so.2(+0x646ba6) [0x7f60e25d8ba6] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 9: make_fcontext() 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout:2024-04-20T15:58:29.360+0000 7f5fc4057700 -1 *** Caught signal (Segmentation fault) ** 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: in thread 7f5fc4057700 thread_name:radosgw 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: ceph version 17.2.7-902-g7f4fa80c (7f4fa80c8c97f53f52b1189d2347b5761f6287e3) quincy (stable) 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f60dfddacf0] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 2: /lib64/libradosgw.so.2(+0x90c5b1) [0x7f60e289e5b1] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 3: (RGWObjManifest::~RGWObjManifest()+0x1d3) [0x7f60e28b5233] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 4: (RGWObjState::~RGWObjState()+0x1e3) [0x7f60e2acf883] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 5: /lib64/libradosgw.so.2(+0x6e46ee) [0x7f60e26766ee] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 6: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, std::shared_ptr, int*)+0x1702) [0x7f60e267a882] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 7: /lib64/libradosgw.so.2(+0x645765) [0x7f60e25d7765] 2024-04-20T15:58:29.360 INFO:tasks.rgw.client.0.smithi071.stdout: 8: /lib64/libradosgw.so.2(+0x646ba6) [0x7f60e25d8ba6] 2024-04-20T15:58:29.361 INFO:tasks.rgw.client.0.smithi071.stdout: 9: make_fcontext() 2024-04-20T15:58:29.361 INFO:tasks.rgw.client.0.smithi071.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-20T15:58:29.361 INFO:tasks.rgw.client.0.smithi071.stdout: 2024-04-20T15:58:29.371 INFO:tasks.rgw.client.0.smithi071.stdout: 0> 2024-04-20T15:58:29.360+0000 7f5fc4057700 -1 *** Caught signal (Segmentation fault) ** 2024-04-20T15:58:29.371 INFO:tasks.rgw.client.0.smithi071.stdout: in thread 7f5fc4057700 thread_name:radosgw 2024-04-20T15:58:29.371 INFO:tasks.rgw.client.0.smithi071.stdout: 2024-04-20T15:58:29.371 INFO:tasks.rgw.client.0.smithi071.stdout: ceph version 17.2.7-902-g7f4fa80c (7f4fa80c8c97f53f52b1189d2347b5761f6287e3) quincy (stable) 2024-04-20T15:58:29.371 INFO:tasks.rgw.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f60dfddacf0] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 2: /lib64/libradosgw.so.2(+0x90c5b1) [0x7f60e289e5b1] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 3: (RGWObjManifest::~RGWObjManifest()+0x1d3) [0x7f60e28b5233] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 4: (RGWObjState::~RGWObjState()+0x1e3) [0x7f60e2acf883] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 5: /lib64/libradosgw.so.2(+0x6e46ee) [0x7f60e26766ee] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 6: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, std::shared_ptr, int*)+0x1702) [0x7f60e267a882] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 7: /lib64/libradosgw.so.2(+0x645765) [0x7f60e25d7765] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 8: /lib64/libradosgw.so.2(+0x646ba6) [0x7f60e25d8ba6] 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 9: make_fcontext() 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-20T15:58:29.372 INFO:tasks.rgw.client.0.smithi071.stdout: 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 0> 2024-04-20T15:58:29.360+0000 7f5fc4057700 -1 *** Caught signal (Segmentation fault) ** 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: in thread 7f5fc4057700 thread_name:radosgw 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: ceph version 17.2.7-902-g7f4fa80c (7f4fa80c8c97f53f52b1189d2347b5761f6287e3) quincy (stable) 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f60dfddacf0] 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 2: /lib64/libradosgw.so.2(+0x90c5b1) [0x7f60e289e5b1] 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 3: (RGWObjManifest::~RGWObjManifest()+0x1d3) [0x7f60e28b5233] 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 4: (RGWObjState::~RGWObjState()+0x1e3) [0x7f60e2acf883] 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 5: /lib64/libradosgw.so.2(+0x6e46ee) [0x7f60e26766ee] 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 6: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, std::shared_ptr, int*)+0x1702) [0x7f60e267a882] 2024-04-20T15:58:29.384 INFO:tasks.rgw.client.0.smithi071.stdout: 7: /lib64/libradosgw.so.2(+0x645765) [0x7f60e25d7765] 2024-04-20T15:58:29.385 INFO:tasks.rgw.client.0.smithi071.stdout: 8: /lib64/libradosgw.so.2(+0x646ba6) [0x7f60e25d8ba6] 2024-04-20T15:58:29.385 INFO:tasks.rgw.client.0.smithi071.stdout: 9: make_fcontext() 2024-04-20T15:58:29.385 INFO:tasks.rgw.client.0.smithi071.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-20T15:58:29.385 INFO:tasks.rgw.client.0.smithi071.stdout: 2024-04-20T15:58:29.470 INFO:tasks.rgw.client.0.smithi071.stderr:daemon-helper: command crashed with signal 11 2024-04-20T15:58:34.684 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2024-04-20T15:58:40.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~6s 2024-04-20T15:58:47.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~13s 2024-04-20T15:58:53.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~19s 2024-04-20T15:58:59.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~25s 2024-04-20T15:59:06.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~32s 2024-04-20T15:59:12.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~38s 2024-04-20T15:59:18.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~44s 2024-04-20T15:59:25.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~50s 2024-04-20T15:59:31.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~57s 2024-04-20T15:59:37.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~63s 2024-04-20T15:59:44.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~69s 2024-04-20T15:59:50.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~76s 2024-04-20T15:59:56.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~82s 2024-04-20T16:00:03.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~88s 2024-04-20T16:00:09.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~95s 2024-04-20T16:00:15.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~101s 2024-04-20T16:00:21.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~107s 2024-04-20T16:00:28.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~114s 2024-04-20T16:00:34.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~120s 2024-04-20T16:00:40.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~126s 2024-04-20T16:00:47.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~133s 2024-04-20T16:00:53.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~139s 2024-04-20T16:00:59.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~145s 2024-04-20T16:01:06.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~151s 2024-04-20T16:01:12.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~158s 2024-04-20T16:01:18.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~164s 2024-04-20T16:01:25.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~170s 2024-04-20T16:01:31.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~177s 2024-04-20T16:01:32.976 INFO:teuthology.orchestra.run.smithi071.stderr:test pushing amqp s3 notification on master ... ERROR 2024-04-20T16:01:37.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~183s 2024-04-20T16:01:43.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~189s 2024-04-20T16:01:50.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~196s 2024-04-20T16:01:56.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~202s 2024-04-20T16:02:02.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~208s 2024-04-20T16:02:09.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~215s 2024-04-20T16:02:15.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~221s 2024-04-20T16:02:21.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~227s 2024-04-20T16:02:28.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~233s 2024-04-20T16:02:34.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~240s 2024-04-20T16:02:40.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~246s 2024-04-20T16:02:47.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~252s 2024-04-20T16:02:53.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~259s 2024-04-20T16:02:56.783 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn.test_ps_s3_creation_triggers_on_master ... ERROR 2024-04-20T16:02:59.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~265s 2024-04-20T16:03:06.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~271s 2024-04-20T16:03:09.906 INFO:teuthology.orchestra.run.smithi071.stderr:test that uploads object using HTTP POST ... /usr/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.18) or chardet (3.0.4) doesn't match a supported version! 2024-04-20T16:03:09.906 INFO:teuthology.orchestra.run.smithi071.stderr: RequestsDependencyWarning) 2024-04-20T16:03:09.906 INFO:teuthology.orchestra.run.smithi071.stderr:/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2024-04-20T16:03:09.906 INFO:teuthology.orchestra.run.smithi071.stderr: warnings.warn(warning, PythonDeprecationWarning) 2024-04-20T16:03:09.906 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR 2024-04-20T16:03:12.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~278s 2024-04-20T16:03:18.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~284s 2024-04-20T16:03:24.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~290s 2024-04-20T16:03:31.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~297s 2024-04-20T16:03:37.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~303s 2024-04-20T16:03:37.588 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-20T16:03:38.896 INFO:tasks.ceph.osd.0:Sent signal 15 2024-04-20T16:03:38.897 INFO:tasks.ceph.osd.1:Sent signal 15 2024-04-20T16:03:38.897 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-20T16:03:38.898 INFO:tasks.ceph.osd.3:Sent signal 15 2024-04-20T16:03:38.898 INFO:tasks.ceph.osd.4:Sent signal 15 2024-04-20T16:03:38.898 INFO:tasks.ceph.osd.5:Sent signal 15 2024-04-20T16:03:38.898 INFO:tasks.ceph.osd.6:Sent signal 15 2024-04-20T16:03:38.898 INFO:tasks.ceph.osd.7:Sent signal 15 2024-04-20T16:03:38.899 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-20T16:03:38.899 INFO:tasks.ceph.mon.c:Sent signal 15 2024-04-20T16:03:38.899 INFO:tasks.ceph.mon.b:Sent signal 15 2024-04-20T16:03:38.899 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-04-20T16:03:38.899 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-20T16:03:38.900 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T16:03:38.900+0000 7fdb51e15700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43936) UID: 0 2024-04-20T16:03:38.900 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T16:03:38.900+0000 7fdb51e15700 -1 osd.4 36 *** Got signal Terminated *** 2024-04-20T16:03:38.900 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-04-20T16:03:38.900+0000 7fdb51e15700 -1 osd.4 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.901 INFO:tasks.ceph.mon.b.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f5375336700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 43283) UID: 0 2024-04-20T16:03:38.901 INFO:tasks.ceph.mon.b.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f5375336700 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-04-20T16:03:38.901 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T16:03:38.897+0000 7f470ec66700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43994) UID: 0 2024-04-20T16:03:38.901 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T16:03:38.897+0000 7f470ec66700 -1 osd.0 36 *** Got signal Terminated *** 2024-04-20T16:03:38.901 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-04-20T16:03:38.897+0000 7f470ec66700 -1 osd.0 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.902 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T16:03:38.900+0000 7fcb24712700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43948) UID: 0 2024-04-20T16:03:38.902 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T16:03:38.900+0000 7fcb24712700 -1 osd.5 36 *** Got signal Terminated *** 2024-04-20T16:03:38.902 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-04-20T16:03:38.900+0000 7fcb24712700 -1 osd.5 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.902 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T16:03:38.898+0000 7fad77949700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43998) UID: 0 2024-04-20T16:03:38.902 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T16:03:38.898+0000 7fad77949700 -1 osd.1 36 *** Got signal Terminated *** 2024-04-20T16:03:38.902 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-04-20T16:03:38.898+0000 7fad77949700 -1 osd.1 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.903 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f7c89afa700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43954) UID: 0 2024-04-20T16:03:38.903 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f7c89afa700 -1 osd.6 36 *** Got signal Terminated *** 2024-04-20T16:03:38.903 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f7c89afa700 -1 osd.6 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.903 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f54f9e78700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43953) UID: 0 2024-04-20T16:03:38.903 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f54f9e78700 -1 osd.7 36 *** Got signal Terminated *** 2024-04-20T16:03:38.903 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-04-20T16:03:38.900+0000 7f54f9e78700 -1 osd.7 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.904 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T16:03:38.898+0000 7f377b835700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 44004) UID: 0 2024-04-20T16:03:38.904 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T16:03:38.898+0000 7f377b835700 -1 osd.2 36 *** Got signal Terminated *** 2024-04-20T16:03:38.904 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-20T16:03:38.898+0000 7f377b835700 -1 osd.2 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.904 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T16:03:38.898+0000 7fad64ba7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 44010) UID: 0 2024-04-20T16:03:38.904 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T16:03:38.898+0000 7fad64ba7700 -1 osd.3 36 *** Got signal Terminated *** 2024-04-20T16:03:38.904 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-20T16:03:38.898+0000 7fad64ba7700 -1 osd.3 36 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T16:03:38.905 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-04-20T16:03:38.899+0000 7fb47275b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 43600) UID: 0 2024-04-20T16:03:38.905 INFO:tasks.ceph.mon.a.smithi071.stderr:2024-04-20T16:03:38.899+0000 7fb47275b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-20T16:03:38.905 INFO:tasks.ceph.mon.c.smithi071.stderr:2024-04-20T16:03:38.899+0000 7f790b452700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 43602) UID: 0 2024-04-20T16:03:38.905 INFO:tasks.ceph.mon.c.smithi071.stderr:2024-04-20T16:03:38.899+0000 7f790b452700 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-04-20T16:03:38.905 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T16:03:38.900+0000 7f04d0596700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 43651) UID: 0 2024-04-20T16:03:38.905 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-04-20T16:03:38.900+0000 7f04d0596700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-20T16:03:39.101 INFO:tasks.ceph.mgr.x.smithi119.stderr:daemon-helper: command crashed with signal 15 2024-04-20T16:04:33.845 INFO:teuthology.orchestra.run.smithi071.stderr:test multipart object upload on master ... ERROR 2024-04-20T16:05:08.606 INFO:teuthology.orchestra.run.smithi071.stderr:test s3 notification of metadata on master ... ERROR 2024-04-20T16:06:15.116 INFO:teuthology.orchestra.run.smithi071.stderr:test s3 notification of metadata on master ... ERROR 2024-04-20T16:06:58.082 INFO:teuthology.orchestra.run.smithi071.stderr:test s3 notification of tags on master ... ERROR 2024-04-20T16:07:42.106 INFO:teuthology.orchestra.run.smithi071.stderr:test s3 notification of object versions ... ERROR 2024-04-20T16:08:12.700 INFO:teuthology.orchestra.run.smithi071.stderr:test s3 notification of deletion markers on master ... ERROR 2024-04-20T16:09:14.363 INFO:teuthology.orchestra.run.smithi071.stderr:test pushing persistent notification amqp ... ERROR 2024-04-20T16:10:03.057 INFO:teuthology.orchestra.run.smithi071.stderr:test pushing persistent notification of large notifications ... ERROR 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test pushing amqp s3 notification on master 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 1088, in test_ps_s3_notification_push_amqp_on_master 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: keys = list(bucket.list()) 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: encoding_type=encoding_type) 2024-04-20T16:10:03.058 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/bucket.py", line 474, in get_all_keys 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: '', headers, **params) 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/bucket.py", line 400, in _get_all 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: query_args=query_args) 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.059 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.060 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/ 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/ 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/ 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.061 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Unk+eK9U4bhXo5LUElQHn6wpjhs= 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Unk+eK9U4bhXo5LUElQHn6wpjhs=', 'Content-Length': '0'} 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002e42070fa8b3abb9-006623e623-11a9-default'), ('Content-Length', '0'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: Using EPollPoller 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.base_connection: DEBUG: Created default connection workflow 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Starting AMQP Connection workflow asynchronously. 2024-04-20T16:10:03.062 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=854.557750681 and callback=functools.partial(>, first=True); now=854.557750681; delay=0 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Beginning a new AMQP connection workflow cycle; attempts remaining after this: 0 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=854.557846701 and callback=>; now=854.557846701; delay=0 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: _try_next_config_async: 'localhost':5672 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: add_callback_threadsafe: added callback=> 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: process_timeouts: invoking callback=> 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: Invoking asynchronous getaddrinfo() completion callback; host='localhost' 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: getaddrinfo returned 2 records 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Attempting to connect using address record (, , 6, '', ('::1', 5672, 0, 0)) 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: Pika version 1.3.1 connecting to ('::1', 5672, 0, 0) 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: add_callback_threadsafe: added callback=> 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=864.558500624 and callback=>; now=854.558500624; delay=10.0 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=869.558532928 and callback=>; now=854.558532928; delay=15.0 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: process_timeouts: invoking callback=> 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.063 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) added handler Wr 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Connection-establishment is in progress for . 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: INFO: Socket connected: 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncSocketConnector._report_completion(None); 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) removed handler 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: remove_timeout: removing timeout with deadline=864.558500624 and callback=> 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: TCP connection to broker established: . 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector.start(); 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: add_callback_threadsafe: added callback=> 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: process_timeouts: invoking callback=> 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._start_async(); 2024-04-20T16:10:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._linkup() 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: CLOSED (prev=CLOSED) 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: INIT (prev=CLOSED) 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: Using external connection workflow. 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncTransportBase.__init__: 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_reader(11, >) 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_reader(11, _) added handler Rd 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _linkup(): created transport 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: PROTOCOL (prev=INIT) 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.065 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _linkup(): introduced transport to protocol ; _StreamingProtocolShim: params=> 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._report_completion((, _StreamingProtocolShim: params=>)); 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: Streaming transport linked up: (, _StreamingProtocolShim: params=>). 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._cleanup(False) 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:Connection.Start 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.066 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>> for "0:Connection.Start" 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: START (prev=PROTOCOL) 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:Connection.Tune 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.067 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>> for "0:Connection.Tune" 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: TUNE (prev=START) 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: Creating a HeartbeatChecker: 60 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: timeout: 60.000000 send_interval: 30.000000 check_interval: 65.000000 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=884.561615928 and callback=>; now=854.561615928; delay=30.0 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=919.561646831 and callback=>; now=854.561646831; delay=65 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.068 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:Connection.OpenOk 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>> for "0:Connection.OpenOk" 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: OPEN (prev=TUNE) 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:_on_connection_open_ok 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "0:_on_connection_open_ok" 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: AMQPConnector: AMQP handshake attempt completed; state=3; error=None; 'localhost'/(, , 6, '', ('::1', 5672, 0, 0)) 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: AMQPConnector: AMQP connection established for 'localhost'/(, , 6, '', ('::1', 5672, 0, 0)): params=> 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: AMQPConnector - reporting success: params=> 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: remove_timeout: removing timeout with deadline=869.558532928 and callback=> 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Connection attempt completed with params=> 2024-04-20T16:10:03.069 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: AMQPConnectionWorkflow - reporting success: params=> 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.blocking_connection: INFO: Connection workflow succeeded: params=> 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: Creating channel 1 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>, 'one_shot': True, 'only': params=>>, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.070 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>>, 'one_shot': False, 'only': params=>>, 'arguments': None} 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.blocking_connection: INFO: Created channel=1 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Channel.OpenOk 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Channel.OpenOk" 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Channel.OpenOk" 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.072 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Exchange.DeclareOk 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.073 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Exchange.DeclareOk" 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Exchange.DeclareOk" 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.074 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Queue.DeclareOk 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Queue.DeclareOk" 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Queue.DeclareOk" 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.075 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Queue.BindOk 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.076 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Queue.BindOk" 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Queue.BindOk" 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[(, {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'})] 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'}, 'calls': 1} 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'}, 'calls': 1} 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Basic.ConsumeOk 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Comparing {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'} to {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'} 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #1: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'}, 'calls': 0} 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Comparing {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'} to {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'} 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #1: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.3b012a5030f64db8a90f971494de5aee'}, 'calls': 0} 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Basic.ConsumeOk" 2024-04-20T16:10:03.078 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Basic.ConsumeOk" 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Basic.ConsumeOk" 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Discarding frame "])> 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: Using EPollPoller 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.base_connection: DEBUG: Created default connection workflow 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Starting AMQP Connection workflow asynchronously. 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=854.569721869 and callback=functools.partial(>, first=True); now=854.569721869; delay=0 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Beginning a new AMQP connection workflow cycle; attempts remaining after this: 0 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=854.56979934 and callback=>; now=854.56979934; delay=0 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: _try_next_config_async: 'localhost':5672 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: add_callback_threadsafe: added callback=> 2024-04-20T16:10:03.079 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: process_timeouts: invoking callback=> 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: Invoking asynchronous getaddrinfo() completion callback; host='localhost' 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: getaddrinfo returned 2 records 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Attempting to connect using address record (, , 6, '', ('::1', 5672, 0, 0)) 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: Pika version 1.3.1 connecting to ('::1', 5672, 0, 0) 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: add_callback_threadsafe: added callback=> 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=864.57039738 and callback=>; now=854.57039738; delay=10.0 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=869.570428899 and callback=>; now=854.570428899; delay=15.0 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: process_timeouts: invoking callback=> 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) added handler Wr 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Connection-establishment is in progress for . 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: INFO: Socket connected: 2024-04-20T16:10:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncSocketConnector._report_completion(None); 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) removed handler 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: remove_timeout: removing timeout with deadline=864.57039738 and callback=> 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: TCP connection to broker established: . 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector.start(); 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: add_callback_threadsafe: added callback=> 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: process_timeouts: invoking callback=> 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._start_async(); 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._linkup() 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: CLOSED (prev=CLOSED) 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.081 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: INIT (prev=CLOSED) 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: Using external connection workflow. 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncTransportBase.__init__: 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_reader(15, >) 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_reader(15, _) added handler Rd 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _linkup(): created transport 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: PROTOCOL (prev=INIT) 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _linkup(): introduced transport to protocol ; _StreamingProtocolShim: params=> 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._report_completion((, _StreamingProtocolShim: params=>)); 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: Streaming transport linked up: (, _StreamingProtocolShim: params=>). 2024-04-20T16:10:03.082 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: _AsyncStreamConnector._cleanup(False) 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:Connection.Start 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>> for "0:Connection.Start" 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: START (prev=PROTOCOL) 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.083 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:Connection.Tune 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>> for "0:Connection.Tune" 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: TUNE (prev=START) 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: Creating a HeartbeatChecker: 60 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: timeout: 60.000000 send_interval: 30.000000 check_interval: 65.000000 2024-04-20T16:10:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=884.573272765 and callback=>; now=854.573272765; delay=30.0 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=919.573303645 and callback=>; now=854.573303645; delay=65 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:Connection.OpenOk 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.085 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>> for "0:Connection.OpenOk" 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: New Connection state: OPEN (prev=TUNE) 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 0:_on_connection_open_ok 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "0:_on_connection_open_ok" 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: AMQPConnector: AMQP handshake attempt completed; state=3; error=None; 'localhost'/(, , 6, '', ('::1', 5672, 0, 0)) 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: AMQPConnector: AMQP connection established for 'localhost'/(, , 6, '', ('::1', 5672, 0, 0)): params=> 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: AMQPConnector - reporting success: params=> 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: remove_timeout: removing timeout with deadline=869.570428899 and callback=> 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: DEBUG: Connection attempt completed with params=> 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.connection_workflow: INFO: AMQPConnectionWorkflow - reporting success: params=> 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.blocking_connection: INFO: Connection workflow succeeded: params=> 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.086 INFO:teuthology.orchestra.run.smithi071.stderr:pika.connection: DEBUG: Creating channel 1 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>, 'one_shot': True, 'only': params=>>, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>>, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>>, 'one_shot': False, 'only': params=>>, 'arguments': None} 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': False, 'only': None, 'arguments': None} 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.blocking_connection: INFO: Created channel=1 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Channel.OpenOk 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Channel.OpenOk" 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Channel.OpenOk" 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Exchange.DeclareOk 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Exchange.DeclareOk" 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Exchange.DeclareOk" 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Queue.DeclareOk 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Queue.DeclareOk" 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Queue.DeclareOk" 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[] 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1} 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Queue.BindOk 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #0: {'callback': >, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0} 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Queue.BindOk" 2024-04-20T16:10:03.093 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Queue.BindOk" 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Entering blocking state on frame ; acceptable_replies=[(, {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'})] 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding on_synchronous_complete callback 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'}, 'calls': 1} 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Adding passed-in RPC response callback 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Added: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'}, 'calls': 1} 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.094 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing 1:Basic.ConsumeOk 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Comparing {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'} to {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'} 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #1: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'}, 'calls': 0} 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Processing use of oneshot callback 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: 0 registered uses left 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Comparing {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'} to {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'} 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Removing callback #1: {'callback': params=>>>, 'one_shot': True, 'only': None, 'arguments': {'consumer_tag': 'ctag1.66822ece01914fd3b0d2df251b731a1c'}, 'calls': 0} 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling > for "1:Basic.ConsumeOk" 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Basic.ConsumeOk" 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: 0 blocked frames 2024-04-20T16:10:03.095 INFO:teuthology.orchestra.run.smithi071.stderr:pika.callback: DEBUG: Calling params=>>> for "1:Basic.ConsumeOk" 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:pika.channel: DEBUG: Discarding frame "])> 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP receiver started 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP receiver started 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.sns: calling handler 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.regions: DEBUG: Creating a regex based endpoint for sns, default 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Setting sns timeout as (60, 60) 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.client: DEBUG: Registering retry handlers for service: sns 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.sns.CreateTopic: calling handler 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-552.el8.x86_64 Botocore/1.26.10'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'lbboah-2_topic_1', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'amqp://localhost', 'Attributes.entry.2.key': 'amqp-exchange', 'Attributes.entry.2.value': 'ex1', 'Attributes.entry.3.key': 'amqp-ack-level', 'Attributes.entry.3.value': 'broker'}, 'url': 'http://smithi071.front.sepia.ceph.com:80/', 'context': {'client_region': 'default', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler > 2024-04-20T16:10:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.sns.CreateTopic: calling handler 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:POST 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:/ 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T155827Z 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:content-type;host;x-amz-date 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:0db7178cd74f60e728e304c4e608e3ceef37d6b63841b9d9983926ebf970a4e4 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T155827Z 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/default/sns/aws4_request 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:683b73e54f0d8b46e49020cbd694c5427b0022e2187db738f0294347d6becbe6 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:c185b308f189c6b9a318d1b7f14650ea1dd1992c5428aafeec394eb24c258c98 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: http://smithi071.front.sepia.ceph.com:80 "POST / HTTP/1.1" 200 312 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000c5c9e90cfea26020-006623e623-11a9-default', 'Content-Type': 'application/xml', 'Content-Length': '312', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.parsers: DEBUG: Response body: 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:b'arn:aws:sns:default::lbboah-2_topic_1f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14252178737811578912' 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.sns.CreateTopic: calling handler 2024-04-20T16:10:03.098 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.sns: calling handler 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.regions: DEBUG: Creating a regex based endpoint for sns, default 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Setting sns timeout as (60, 60) 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.client: DEBUG: Registering retry handlers for service: sns 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.sns.CreateTopic: calling handler 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.sns.CreateTopic: calling handler 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateTopic) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-552.el8.x86_64 Botocore/1.26.10'}, 'body': {'Action': 'CreateTopic', 'Version': '2010-03-31', 'Name': 'lbboah-2_topic_2', 'Attributes.entry.1.key': 'push-endpoint', 'Attributes.entry.1.value': 'amqp://localhost', 'Attributes.entry.2.key': 'amqp-exchange', 'Attributes.entry.2.value': 'ex1', 'Attributes.entry.3.key': 'amqp-ack-level', 'Attributes.entry.3.value': 'routable'}, 'url': 'http://smithi071.front.sepia.ceph.com:80/', 'context': {'client_region': 'default', 'client_config': , 'has_streaming_input': False, 'auth_type': None}} 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler > 2024-04-20T16:10:03.099 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.sns.CreateTopic: calling handler 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:POST 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:/ 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:content-type:application/x-www-form-urlencoded; charset=utf-8 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T155827Z 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:content-type;host;x-amz-date 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:a73a0ac639000e1de31c13cedc69361073c5b7f97fa79664e62efd850e3dbf30 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.100 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T155827Z 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/default/sns/aws4_request 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:19ea444120c8c3b1f0e221fba792c698c1efb9330016c5040e33c08e2f097bc3 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:5514f02438185b4e12b74d9ef4a124cb30f95c6ed150dfaa8072861db1cf9eae 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.sns.CreateTopic: calling handler 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: http://smithi071.front.sepia.ceph.com:80 "POST / HTTP/1.1" 200 312 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000c5327125507dfe74-006623e623-11a9-default', 'Content-Type': 'application/xml', 'Content-Length': '312', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.parsers: DEBUG: Response body: 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:b'arn:aws:sns:default::lbboah-2_topic_2f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14209544179385499252' 2024-04-20T16:10:03.101 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.sns.CreateTopic: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7fd0cea10620> 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.102 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=PutBucketNotificationConfiguration) with params: {'url_path': '/lbboah-2?notification', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-552.el8.x86_64 Botocore/1.26.10'}, 'body': b'lbboah-2_notif_1arn:aws:sns:default::lbboah-2_topic_1lbboah-2_notif_2arn:aws:sns:default::lbboah-2_topic_2s3:ObjectCreated:*', 'url': 'http://smithi071.front.sepia.ceph.com:80/lbboah-2?notification', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'lbboah-2'}}} 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:notification= 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.103 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-content-sha256:6a96f9d6a00313f73e46dfe73f883840ffef2d96fb2b809f5e5e978620c552c9 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T155827Z 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:host;x-amz-content-sha256;x-amz-date 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:6a96f9d6a00313f73e46dfe73f883840ffef2d96fb2b809f5e5e978620c552c9 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T155827Z 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/us-east-1/s3/aws4_request 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:51de2f1d56d5c31ea9a72b57630fb4651a80f480113ea8a0da95c0743211e667 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:8108b132a237a2038dbf3d153812d57a092e80e93be615e3e4458a13fbf33875 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.104 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: http://smithi071.front.sepia.ceph.com:80 "PUT /lbboah-2?notification HTTP/1.1" 200 0 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000aabbbdc6ec646cd7-006623e623-11a9-default', 'Content-Length': '0', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Connection': 'Keep-Alive'} 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.parsers: DEBUG: Response body: 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:b'' 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.PutBucketNotificationConfiguration: calling handler > 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/0 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/0 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/0 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '2ZC5uW4ORYUmbw/Ga9GioQ==', 'Content-Length': '3011644', 'Expect': '100-Continue'} 2024-04-20T16:10:03.105 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:2ZC5uW4ORYUmbw/Ga9GioQ== 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/0 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PvdPFqQpdBhzZIalCXIgFe+tJ98= 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '2ZC5uW4ORYUmbw/Ga9GioQ==', 'Content-Length': '3011644', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PvdPFqQpdBhzZIalCXIgFe+tJ98='} 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/1 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/1 2024-04-20T16:10:03.106 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/1 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TiqUvI1Uf/4Kq4LnLQmc0Q==', 'Content-Length': '3010944', 'Expect': '100-Continue'} 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:TiqUvI1Uf/4Kq4LnLQmc0Q== 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/1 2024-04-20T16:10:03.107 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:xqjroau0D6gr+J9aWpctYjs8I1Y= 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TiqUvI1Uf/4Kq4LnLQmc0Q==', 'Content-Length': '3010944', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:xqjroau0D6gr+J9aWpctYjs8I1Y='} 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/2 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/2 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/2 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'zt7sWbsSa/kFUms56/sUWA==', 'Content-Length': '3011032', 'Expect': '100-Continue'} 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.108 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:zt7sWbsSa/kFUms56/sUWA== 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/2 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:J0OXDxNcEeHn+Wnidz6tDj0zyB4= 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'zt7sWbsSa/kFUms56/sUWA==', 'Content-Length': '3011032', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:J0OXDxNcEeHn+Wnidz6tDj0zyB4='} 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/3 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/3 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/3 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'g+0pK/76bsWN26J+TzQdiQ==', 'Content-Length': '3010626', 'Expect': '100-Continue'} 2024-04-20T16:10:03.109 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:g+0pK/76bsWN26J+TzQdiQ== 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/3 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:fPX4MUlk4wC2uYTiUkzggc2my1I= 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'g+0pK/76bsWN26J+TzQdiQ==', 'Content-Length': '3010626', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:fPX4MUlk4wC2uYTiUkzggc2my1I='} 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/4 2024-04-20T16:10:03.110 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/4 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/4 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JRmasu/dqVaQ6+jGsnnFZw==', 'Content-Length': '3006253', 'Expect': '100-Continue'} 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:JRmasu/dqVaQ6+jGsnnFZw== 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.111 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/4 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3lgyK+RPsPmYMOkF67E9LxWV+hI= 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JRmasu/dqVaQ6+jGsnnFZw==', 'Content-Length': '3006253', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3lgyK+RPsPmYMOkF67E9LxWV+hI='} 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.274120Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8442162171128862482","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"0","size":3011644,"eTag":"d990b9b96e0e4585266f0fc66bd1a2a1","versionId":"","sequencer":"23E62366A7E55D11","metadata":[],"tags":[]}},"eventId":"1713628707.291366.d990b9b96e0e4585266f0fc66bd1a2a1","opaqueData":""}]}' 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.305441Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4408914129341742653","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"1","size":3010944,"eTag":"4e2a94bc8d547ffe0aab82e72d099cd1","versionId":"","sequencer":"23E62366D794E812","metadata":[],"tags":[]}},"eventId":"1713628707.317232.4e2a94bc8d547ffe0aab82e72d099cd1","opaqueData":""}]}' 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/5 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/5 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/5 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ZJGVVHC4f6Fm+auFPaPwdA==', 'Content-Length': '3009526', 'Expect': '100-Continue'} 2024-04-20T16:10:03.112 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:ZJGVVHC4f6Fm+auFPaPwdA== 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/5 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:x254MJQG0F5IQFzyYeuEyNiPvJE= 2024-04-20T16:10:03.113 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ZJGVVHC4f6Fm+auFPaPwdA==', 'Content-Length': '3009526', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:x254MJQG0F5IQFzyYeuEyNiPvJE='} 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/6 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/6 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/6 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Dt8me0+gumE8OlTCB7yhmQ==', 'Content-Length': '3009541', 'Expect': '100-Continue'} 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:Dt8me0+gumE8OlTCB7yhmQ== 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/6 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lYLgcf8byUS1W6ptl1qKxBDzYEM= 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Dt8me0+gumE8OlTCB7yhmQ==', 'Content-Length': '3009541', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lYLgcf8byUS1W6ptl1qKxBDzYEM='} 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/7 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/7 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/7 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '5JCbf0gQd/PXAjoz0S4krA==', 'Content-Length': '3009682', 'Expect': '100-Continue'} 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.115 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:5JCbf0gQd/PXAjoz0S4krA== 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/7 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:1lscsaagAk/hlZzA6uUIVdxOlcE= 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '5JCbf0gQd/PXAjoz0S4krA==', 'Content-Length': '3009682', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:1lscsaagAk/hlZzA6uUIVdxOlcE='} 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/8 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/8 2024-04-20T16:10:03.116 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/8 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '0XmNBsMVprdlVDwKT7hoWQ==', 'Content-Length': '3009394', 'Expect': '100-Continue'} 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:0XmNBsMVprdlVDwKT7hoWQ== 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/8 2024-04-20T16:10:03.117 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cMh4ePYG8rf6KUz07YijMI++E6w= 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '0XmNBsMVprdlVDwKT7hoWQ==', 'Content-Length': '3009394', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cMh4ePYG8rf6KUz07YijMI++E6w='} 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/9 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/9 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/9 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uJzhDwuASBe8VSd+4CRqeg==', 'Content-Length': '3009179', 'Expect': '100-Continue'} 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.118 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:uJzhDwuASBe8VSd+4CRqeg== 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/9 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/mrAiQTrzJvtZygmnWc9zbysFSk= 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uJzhDwuASBe8VSd+4CRqeg==', 'Content-Length': '3009179', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/mrAiQTrzJvtZygmnWc9zbysFSk='} 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.336882Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.5836218460231399929","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"2","size":3011032,"eTag":"cedeec59bb126bf905526b39ebfb1458","versionId":"","sequencer":"23E62366528F0A15","metadata":[],"tags":[]}},"eventId":"1713628707.353013.cedeec59bb126bf905526b39ebfb1458","opaqueData":""}]}' 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.378579Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10266198110865559058","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"3","size":3010626,"eTag":"83ed292bfefa6ec58ddba27e4f341d89","versionId":"","sequencer":"23E623669D902C17","metadata":[],"tags":[]}},"eventId":"1713628707.388796.83ed292bfefa6ec58ddba27e4f341d89","opaqueData":""}]}' 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.119 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.409448Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4091262087918334552","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"4","size":3006253,"eTag":"25199ab2efdda95690ebe8c6b279c567","versionId":"","sequencer":"23E6236607262419","metadata":[],"tags":[]}},"eventId":"1713628707.421799.25199ab2efdda95690ebe8c6b279c567","opaqueData":""}]}' 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/10 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/10 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/10 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'mISusYOY5CUq4EVMoFIMnw==', 'Content-Length': '3010475', 'Expect': '100-Continue'} 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.120 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:mISusYOY5CUq4EVMoFIMnw== 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/10 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m8SPB4A5yCskjqeH38w59Zj2Z7I= 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'mISusYOY5CUq4EVMoFIMnw==', 'Content-Length': '3010475', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m8SPB4A5yCskjqeH38w59Zj2Z7I='} 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/11 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/11 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/11 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7s3ybUwkvvqoKzo7+jFSrw==', 'Content-Length': '3007580', 'Expect': '100-Continue'} 2024-04-20T16:10:03.121 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:7s3ybUwkvvqoKzo7+jFSrw== 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/11 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:ps28JnVxmifOCCyGr5Z3lou8Oag= 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7s3ybUwkvvqoKzo7+jFSrw==', 'Content-Length': '3007580', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:ps28JnVxmifOCCyGr5Z3lou8Oag='} 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/12 2024-04-20T16:10:03.122 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/12 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/12 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'gJdgmC/mNNmhUB6lxJ8eUw==', 'Content-Length': '3010143', 'Expect': '100-Continue'} 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:gJdgmC/mNNmhUB6lxJ8eUw== 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.123 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/12 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KlFxe/Nsc6+PziJSAYmUs/u9G5I= 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'gJdgmC/mNNmhUB6lxJ8eUw==', 'Content-Length': '3010143', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KlFxe/Nsc6+PziJSAYmUs/u9G5I='} 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/13 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/13 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/13 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'cOhVSbWEc52BGVPkjuw1mg==', 'Content-Length': '3012507', 'Expect': '100-Continue'} 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.124 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:cOhVSbWEc52BGVPkjuw1mg== 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/13 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ff+F+2R48OwqOBU5tx45uQSWaiE= 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'cOhVSbWEc52BGVPkjuw1mg==', 'Content-Length': '3012507', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ff+F+2R48OwqOBU5tx45uQSWaiE='} 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.274120Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8442162171128862482","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"0","size":3011644,"eTag":"d990b9b96e0e4585266f0fc66bd1a2a1","versionId":"","sequencer":"23E62366EC295F1A","metadata":[],"tags":[]}},"eventId":"1713628707.442444.d990b9b96e0e4585266f0fc66bd1a2a1","opaqueData":""}]}' 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.125 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.452705Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7237039916465552313","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"5","size":3009526,"eTag":"6491955470b87fa166f9ab853da3f074","versionId":"","sequencer":"23E62366DBAD2A1C","metadata":[],"tags":[]}},"eventId":"1713628707.472559.6491955470b87fa166f9ab853da3f074","opaqueData":""}]}' 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.305441Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4408914129341742653","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"1","size":3010944,"eTag":"4e2a94bc8d547ffe0aab82e72d099cd1","versionId":"","sequencer":"23E6236629625F1A","metadata":[],"tags":[]}},"eventId":"1713628707.442458.4e2a94bc8d547ffe0aab82e72d099cd1","opaqueData":""}]}' 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.484315Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14458501091543955217","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"6","size":3009541,"eTag":"0edf267b4fa0ba613c3a54c207bca199","versionId":"","sequencer":"23E6236627C07A1D","metadata":[],"tags":[]}},"eventId":"1713628707.494583.0edf267b4fa0ba613c3a54c207bca199","opaqueData":""}]}' 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/14 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/14 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/14 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lz5T2kRXb49v4PDSfY877A==', 'Content-Length': '3010215', 'Expect': '100-Continue'} 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.126 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:lz5T2kRXb49v4PDSfY877A== 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/14 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:gsGRXhWqcAw8fvWvU29spGXiqIg= 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lz5T2kRXb49v4PDSfY877A==', 'Content-Length': '3010215', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:gsGRXhWqcAw8fvWvU29spGXiqIg='} 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/15 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/15 2024-04-20T16:10:03.127 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/15 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'UR9qRFcRaLZgC3hsHKpzmw==', 'Content-Length': '3008242', 'Expect': '100-Continue'} 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:UR9qRFcRaLZgC3hsHKpzmw== 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/15 2024-04-20T16:10:03.128 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:9J/L/FnvfwtnKdHUK+QMrymiYSg= 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'UR9qRFcRaLZgC3hsHKpzmw==', 'Content-Length': '3008242', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:9J/L/FnvfwtnKdHUK+QMrymiYSg='} 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/16 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/16 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/16 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3qcf07w1/ctHykrzM8DoCQ==', 'Content-Length': '3011866', 'Expect': '100-Continue'} 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:3qcf07w1/ctHykrzM8DoCQ== 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/16 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:ACiBrg5rlGnGYdcJj/WuPJsvzbE= 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3qcf07w1/ctHykrzM8DoCQ==', 'Content-Length': '3011866', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:ACiBrg5rlGnGYdcJj/WuPJsvzbE='} 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/17 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/17 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/17 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.130 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '60FF7CfCWFP/adrY2+KOVA==', 'Content-Length': '3010090', 'Expect': '100-Continue'} 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:60FF7CfCWFP/adrY2+KOVA== 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/17 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:i6N34S9S/T+pP+a3Glgs6HZmawI= 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '60FF7CfCWFP/adrY2+KOVA==', 'Content-Length': '3010090', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:i6N34S9S/T+pP+a3Glgs6HZmawI='} 2024-04-20T16:10:03.131 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/18 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/18 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/18 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lSAFVb4GmSPSKzFX1N6yyA==', 'Content-Length': '3009133', 'Expect': '100-Continue'} 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:lSAFVb4GmSPSKzFX1N6yyA== 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.132 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/18 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:AdrICdnDW6nt/3d1cItgDB9QnqQ= 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lSAFVb4GmSPSKzFX1N6yyA==', 'Content-Length': '3009133', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:AdrICdnDW6nt/3d1cItgDB9QnqQ='} 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"4e2a94bc8d547ffe0aab82e72d099cd1"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d2f9c0a8c9d7e3d-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.409448Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4091262087918334552","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"4","size":3006253,"eTag":"25199ab2efdda95690ebe8c6b279c567","versionId":"","sequencer":"23E6236622DA5720","metadata":[],"tags":[]}},"eventId":"1713628707.542628.25199ab2efdda95690ebe8c6b279c567","opaqueData":""}]}' 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.336882Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.5836218460231399929","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"2","size":3011032,"eTag":"cedeec59bb126bf905526b39ebfb1458","versionId":"","sequencer":"23E62366DA8A5720","metadata":[],"tags":[]}},"eventId":"1713628707.542608.cedeec59bb126bf905526b39ebfb1458","opaqueData":""}]}' 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.133 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.378579Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10266198110865559058","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"3","size":3010626,"eTag":"83ed292bfefa6ec58ddba27e4f341d89","versionId":"","sequencer":"23E62366298D5720","metadata":[],"tags":[]}},"eventId":"1713628707.542608.83ed292bfefa6ec58ddba27e4f341d89","opaqueData":""}]}' 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d990b9b96e0e4585266f0fc66bd1a2a1"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000752895c422605712-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.530250Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16649186861805254084","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"7","size":3009682,"eTag":"e4909b7f481077f3d7023a33d12e24ac","versionId":"","sequencer":"23E623669F383220","metadata":[],"tags":[]}},"eventId":"1713628707.540162.e4909b7f481077f3d7023a33d12e24ac","opaqueData":""}]}' 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.559480Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16620623409253248390","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"8","size":3009394,"eTag":"d1798d06c315a6b765543c0a4fb86859","versionId":"","sequencer":"23E62366FC14F121","metadata":[],"tags":[]}},"eventId":"1713628707.569447.d1798d06c315a6b765543c0a4fb86859","opaqueData":""}]}' 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.591848Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10419214353645128705","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"9","size":3009179,"eTag":"b89ce10f0b804817bc55277ee0246a7a","versionId":"","sequencer":"23E623665FB1CE23","metadata":[],"tags":[]}},"eventId":"1713628707.600748.b89ce10f0b804817bc55277ee0246a7a","opaqueData":""}]}' 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/19 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/19 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/19 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.134 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TgynwUVjPb+WtIB5BST2Qw==', 'Content-Length': '3009477', 'Expect': '100-Continue'} 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:TgynwUVjPb+WtIB5BST2Qw== 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/19 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:rYAS8pbGlJKov3OvokV5mAS+tqI= 2024-04-20T16:10:03.135 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TgynwUVjPb+WtIB5BST2Qw==', 'Content-Length': '3009477', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:rYAS8pbGlJKov3OvokV5mAS+tqI='} 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/20 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/20 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/20 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '9Gj4HbjwBV4BLvPKQXZ/dg==', 'Content-Length': '3010802', 'Expect': '100-Continue'} 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:9Gj4HbjwBV4BLvPKQXZ/dg== 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.136 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/20 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EMEjHQe5C59u97ggaogSGWYikjg= 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '9Gj4HbjwBV4BLvPKQXZ/dg==', 'Content-Length': '3010802', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EMEjHQe5C59u97ggaogSGWYikjg='} 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/21 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/21 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/21 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '2GT+S4PywibmqruuhVpUgw==', 'Content-Length': '3009895', 'Expect': '100-Continue'} 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.137 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:2GT+S4PywibmqruuhVpUgw== 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/21 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:ApDf1+6G593is6UQklrWdwLpaQs= 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '2GT+S4PywibmqruuhVpUgw==', 'Content-Length': '3009895', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:ApDf1+6G593is6UQklrWdwLpaQs='} 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/23 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/23 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/23 2024-04-20T16:10:03.138 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YSgckYd7nTPMzsEF3XCqdA==', 'Content-Length': '3010537', 'Expect': '100-Continue'} 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:YSgckYd7nTPMzsEF3XCqdA== 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/23 2024-04-20T16:10:03.139 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:t5lEJ/3V+rSwmIkZSGD0cWIvaKk= 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YSgckYd7nTPMzsEF3XCqdA==', 'Content-Length': '3010537', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:t5lEJ/3V+rSwmIkZSGD0cWIvaKk='} 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/22 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/22 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/22 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vZzqvTIt2LjnzSrzsbv0Kg==', 'Content-Length': '3011758', 'Expect': '100-Continue'} 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.140 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:vZzqvTIt2LjnzSrzsbv0Kg== 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/22 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:rFon4yQ+5kUmCbQNu5JSRqLYX4s= 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vZzqvTIt2LjnzSrzsbv0Kg==', 'Content-Length': '3011758', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:rFon4yQ+5kUmCbQNu5JSRqLYX4s='} 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"25199ab2efdda95690ebe8c6b279c567"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038c7153149653258-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"83ed292bfefa6ec58ddba27e4f341d89"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e78dcca07ed8e12-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"cedeec59bb126bf905526b39ebfb1458"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050fe69c499ab79f9-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.141 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.635763Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1926244166498049433","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"10","size":3010475,"eTag":"9884aeb18398e4252ae0454ca0520c9f","versionId":"","sequencer":"23E623666785CF26","metadata":[],"tags":[]}},"eventId":"1713628707.651134.9884aeb18398e4252ae0454ca0520c9f","opaqueData":""}]}' 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.668974Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17338214327827931174","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"11","size":3007580,"eTag":"eecdf26d4c24befaa82b3a3bfa3152af","versionId":"","sequencer":"23E623660169BB28","metadata":[],"tags":[]}},"eventId":"1713628707.683370.eecdf26d4c24befaa82b3a3bfa3152af","opaqueData":""}]}' 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.703076Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.5749624391434700969","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"13","size":3012507,"eTag":"70e85549b584739d811953e48eec359a","versionId":"","sequencer":"23E623663429A02A","metadata":[],"tags":[]}},"eventId":"1713628707.715139.70e85549b584739d811953e48eec359a","opaqueData":""}]}' 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.714900Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.3586888870003489592","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"12","size":3010143,"eTag":"809760982fe634d9a1501ea5c49f1e53","versionId":"","sequencer":"23E62366555D9A2B","metadata":[],"tags":[]}},"eventId":"1713628707.731536.809760982fe634d9a1501ea5c49f1e53","opaqueData":""}]}' 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.452705Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7237039916465552313","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"5","size":3009526,"eTag":"6491955470b87fa166f9ab853da3f074","versionId":"","sequencer":"23E62366E1A14F26","metadata":[],"tags":[]}},"eventId":"1713628707.642752.6491955470b87fa166f9ab853da3f074","opaqueData":""}]}' 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.484315Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14458501091543955217","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"6","size":3009541,"eTag":"0edf267b4fa0ba613c3a54c207bca199","versionId":"","sequencer":"23E6236637A14F26","metadata":[],"tags":[]}},"eventId":"1713628707.642752.0edf267b4fa0ba613c3a54c207bca199","opaqueData":""}]}' 2024-04-20T16:10:03.142 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/24 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/24 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/24 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'FmCDFy5cvZxsENW/5TtqLQ==', 'Content-Length': '3011330', 'Expect': '100-Continue'} 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:FmCDFy5cvZxsENW/5TtqLQ== 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.143 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/24 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:YwbPf6vAAWE0RDBk0cWEWHOXlks= 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'FmCDFy5cvZxsENW/5TtqLQ==', 'Content-Length': '3011330', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:YwbPf6vAAWE0RDBk0cWEWHOXlks='} 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/25 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/25 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/25 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YFhxdZCrjAUWNYgFM2JdgQ==', 'Content-Length': '3012033', 'Expect': '100-Continue'} 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.144 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:YFhxdZCrjAUWNYgFM2JdgQ== 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/25 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Hdomp5bNl0KcHtXm+tywM9tqiB8= 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YFhxdZCrjAUWNYgFM2JdgQ==', 'Content-Length': '3012033', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Hdomp5bNl0KcHtXm+tywM9tqiB8='} 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/26 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/26 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/26 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.145 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'EvhTqlfnK+Wj3Mpw2NfBeg==', 'Content-Length': '3011903', 'Expect': '100-Continue'} 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:EvhTqlfnK+Wj3Mpw2NfBeg== 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/26 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:pTMlcGsi+j1Qw2pMU3mD3q2TLpg= 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'EvhTqlfnK+Wj3Mpw2NfBeg==', 'Content-Length': '3011903', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:pTMlcGsi+j1Qw2pMU3mD3q2TLpg='} 2024-04-20T16:10:03.146 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/27 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/27 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/27 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lDvEf2eD/C/WHlat4m5QYQ==', 'Content-Length': '3011781', 'Expect': '100-Continue'} 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:lDvEf2eD/C/WHlat4m5QYQ== 2024-04-20T16:10:03.147 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/27 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:61xxbEnSD2jrSFHInbyQpAqgNB4= 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lDvEf2eD/C/WHlat4m5QYQ==', 'Content-Length': '3011781', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:61xxbEnSD2jrSFHInbyQpAqgNB4='} 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/28 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/28 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/28 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'MIEU051PHxW9k4nWPcJkYw==', 'Content-Length': '3009833', 'Expect': '100-Continue'} 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.148 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:MIEU051PHxW9k4nWPcJkYw== 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/28 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/1CJq2+3rNvoUgRaBRLQV1P61HQ= 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'MIEU051PHxW9k4nWPcJkYw==', 'Content-Length': '3009833', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/1CJq2+3rNvoUgRaBRLQV1P61HQ='} 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.779043Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15430010947561566927","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"14","size":3010215,"eTag":"973e53da44576f8f6fe0f0d27d8f3bec","versionId":"","sequencer":"23E62366071F7E2F","metadata":[],"tags":[]}},"eventId":"1713628707.796794.973e53da44576f8f6fe0f0d27d8f3bec","opaqueData":""}]}' 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.149 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.810908Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12009946170573655424","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"15","size":3008242,"eTag":"511f6a44571168b6600b786c1caa739b","versionId":"","sequencer":"23E62366A4850431","metadata":[],"tags":[]}},"eventId":"1713628707.822379.511f6a44571168b6600b786c1caa739b","opaqueData":""}]}' 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6491955470b87fa166f9ab853da3f074"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000646f2175e6407bb9-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"e4909b7f481077f3d7023a33d12e24ac"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e70dcb3d5900b5c4-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d1798d06c315a6b765543c0a4fb86859"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6a850ee4e14ed86-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"0edf267b4fa0ba613c3a54c207bca199"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8a6ea236a133711-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.530250Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16649186861805254084","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"7","size":3009682,"eTag":"e4909b7f481077f3d7023a33d12e24ac","versionId":"","sequencer":"23E623664B13482C","metadata":[],"tags":[]}},"eventId":"1713628707.742921.e4909b7f481077f3d7023a33d12e24ac","opaqueData":""}]}' 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.559480Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16620623409253248390","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"8","size":3009394,"eTag":"d1798d06c315a6b765543c0a4fb86859","versionId":"","sequencer":"23E623667AD1482C","metadata":[],"tags":[]}},"eventId":"1713628707.742969.d1798d06c315a6b765543c0a4fb86859","opaqueData":""}]}' 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.591848Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10419214353645128705","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"9","size":3009179,"eTag":"b89ce10f0b804817bc55277ee0246a7a","versionId":"","sequencer":"23E62366933B492C","metadata":[],"tags":[]}},"eventId":"1713628707.742996.b89ce10f0b804817bc55277ee0246a7a","opaqueData":""}]}' 2024-04-20T16:10:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"b89ce10f0b804817bc55277ee0246a7a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090987c4053ab4c01-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/30 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/30 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/30 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Jm6tdYCSINgY9ShsCwPDPg==', 'Content-Length': '3012542', 'Expect': '100-Continue'} 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:Jm6tdYCSINgY9ShsCwPDPg== 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.151 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/30 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:A0JtXt9+GqhFA0CX9S+DOeTX2MQ= 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Jm6tdYCSINgY9ShsCwPDPg==', 'Content-Length': '3012542', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:A0JtXt9+GqhFA0CX9S+DOeTX2MQ='} 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/29 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/29 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/29 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CRX6uCZzKnPR3pL9zKSVhg==', 'Content-Length': '3011722', 'Expect': '100-Continue'} 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.152 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:CRX6uCZzKnPR3pL9zKSVhg== 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/29 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:UMEqwOEakVSNfTfF/8bJ1Iwdjn8= 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CRX6uCZzKnPR3pL9zKSVhg==', 'Content-Length': '3011722', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:UMEqwOEakVSNfTfF/8bJ1Iwdjn8='} 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/32 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/32 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/32 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'kEogrGJ2O+qzz8sBxIYYlg==', 'Content-Length': '3011609', 'Expect': '100-Continue'} 2024-04-20T16:10:03.153 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:kEogrGJ2O+qzz8sBxIYYlg== 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/32 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:qjebSRGgiBSLELOWukZdTHlxuHA= 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'kEogrGJ2O+qzz8sBxIYYlg==', 'Content-Length': '3011609', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:qjebSRGgiBSLELOWukZdTHlxuHA='} 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.154 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.828760Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7756737128526326364","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"16","size":3011866,"eTag":"dea71fd3bc35fdcb47ca4af333c0e809","versionId":"","sequencer":"23E62366F8352732","metadata":[],"tags":[]}},"eventId":"1713628707.841430.dea71fd3bc35fdcb47ca4af333c0e809","opaqueData":""}]}' 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.873935Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13341218863965237301","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"17","size":3010090,"eTag":"eb4145ec27c25853ff69dad8dbe28e54","versionId":"","sequencer":"23E623662EDCD934","metadata":[],"tags":[]}},"eventId":"1713628707.886692.eb4145ec27c25853ff69dad8dbe28e54","opaqueData":""}]}' 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.906261Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15533824819609236920","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"18","size":3009133,"eTag":"95200555be069923d22b3157d4deb2c8","versionId":"","sequencer":"23E62366C3E20F37","metadata":[],"tags":[]}},"eventId":"1713628707.923787.95200555be069923d22b3157d4deb2c8","opaqueData":""}]}' 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.906279Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9699874087643832644","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"19","size":3009477,"eTag":"4e0ca7c145633dbf96b480790524f643","versionId":"","sequencer":"23E623669F7C3337","metadata":[],"tags":[]}},"eventId":"1713628707.926121.4e0ca7c145633dbf96b480790524f643","opaqueData":""}]}' 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.703076Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.5749624391434700969","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"13","size":3012507,"eTag":"70e85549b584739d811953e48eec359a","versionId":"","sequencer":"23E623662E254932","metadata":[],"tags":[]}},"eventId":"1713628707.843654.70e85549b584739d811953e48eec359a","opaqueData":""}]}' 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.714900Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.3586888870003489592","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"12","size":3010143,"eTag":"809760982fe634d9a1501ea5c49f1e53","versionId":"","sequencer":"23E623667A454932","metadata":[],"tags":[]}},"eventId":"1713628707.843662.809760982fe634d9a1501ea5c49f1e53","opaqueData":""}]}' 2024-04-20T16:10:03.155 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.635763Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1926244166498049433","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"10","size":3010475,"eTag":"9884aeb18398e4252ae0454ca0520c9f","versionId":"","sequencer":"23E6236617934832","metadata":[],"tags":[]}},"eventId":"1713628707.843617.9884aeb18398e4252ae0454ca0520c9f","opaqueData":""}]}' 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.810908Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12009946170573655424","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"15","size":3008242,"eTag":"511f6a44571168b6600b786c1caa739b","versionId":"","sequencer":"23E6236686B94932","metadata":[],"tags":[]}},"eventId":"1713628707.843692.511f6a44571168b6600b786c1caa739b","opaqueData":""}]}' 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.668974Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17338214327827931174","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"11","size":3007580,"eTag":"eecdf26d4c24befaa82b3a3bfa3152af","versionId":"","sequencer":"23E623666B234932","metadata":[],"tags":[]}},"eventId":"1713628707.843653.eecdf26d4c24befaa82b3a3bfa3152af","opaqueData":""}]}' 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.779043Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15430010947561566927","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"14","size":3010215,"eTag":"973e53da44576f8f6fe0f0d27d8f3bec","versionId":"","sequencer":"23E623663DAD4932","metadata":[],"tags":[]}},"eventId":"1713628707.843689.973e53da44576f8f6fe0f0d27d8f3bec","opaqueData":""}]}' 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/31 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/31 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/31 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qdJnEychMLlmNlUoTokxjA==', 'Content-Length': '3009651', 'Expect': '100-Continue'} 2024-04-20T16:10:03.156 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:qdJnEychMLlmNlUoTokxjA== 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/31 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:LvqCfXqYUO4GnjAVQwWW1p0NNxs= 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qdJnEychMLlmNlUoTokxjA==', 'Content-Length': '3009651', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:LvqCfXqYUO4GnjAVQwWW1p0NNxs='} 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/33 2024-04-20T16:10:03.157 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/33 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/33 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'xMzgH41EMj+qCYUznsetpg==', 'Content-Length': '3010798', 'Expect': '100-Continue'} 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:xMzgH41EMj+qCYUznsetpg== 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/33 2024-04-20T16:10:03.158 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:xMCCEoUyU0rNfoQ/kVAzRMC71Lk= 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'xMzgH41EMj+qCYUznsetpg==', 'Content-Length': '3010798', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:xMCCEoUyU0rNfoQ/kVAzRMC71Lk='} 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/34 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/34 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/34 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YCKm793IvwRa2KmlChu6Tg==', 'Content-Length': '3010350', 'Expect': '100-Continue'} 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.159 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:YCKm793IvwRa2KmlChu6Tg== 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/34 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:GqIYkZwPWo2mZE6u7yztGv+Uw0E= 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'YCKm793IvwRa2KmlChu6Tg==', 'Content-Length': '3010350', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:27 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:GqIYkZwPWo2mZE6u7yztGv+Uw0E='} 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/35 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/35 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/35 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vy4F13zCohKjefeJ5uoxKw==', 'Content-Length': '3010488', 'Expect': '100-Continue'} 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.160 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:vy4F13zCohKjefeJ5uoxKw== 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/35 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:dU0fN7ljwOOUQQnB+WUyX31LHjc= 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vy4F13zCohKjefeJ5uoxKw==', 'Content-Length': '3010488', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:dU0fN7ljwOOUQQnB+WUyX31LHjc='} 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/36 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/36 2024-04-20T16:10:03.161 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/36 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'DJD08a+26BPZltCLXbVE4w==', 'Content-Length': '3008755', 'Expect': '100-Continue'} 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:DJD08a+26BPZltCLXbVE4w== 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/36 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:E16jhQOKJM2kF4HlUOMf3wA5nRk= 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'DJD08a+26BPZltCLXbVE4w==', 'Content-Length': '3008755', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:E16jhQOKJM2kF4HlUOMf3wA5nRk='} 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"973e53da44576f8f6fe0f0d27d8f3bec"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d622693481cbb2cf-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"eecdf26d4c24befaa82b3a3bfa3152af"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f09db611c1c5a026-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"70e85549b584739d811953e48eec359a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fcac4ebd36308a9-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"809760982fe634d9a1501ea5c49f1e53"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031c7306c515c3b38-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"511f6a44571168b6600b786c1caa739b"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6abe669249b0d80-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"9884aeb18398e4252ae0454ca0520c9f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001abb64dd1ec8dd99-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.163 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.828760Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7756737128526326364","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"16","size":3011866,"eTag":"dea71fd3bc35fdcb47ca4af333c0e809","versionId":"","sequencer":"24E6236663399F02","metadata":[],"tags":[]}},"eventId":"1713628708.043989.dea71fd3bc35fdcb47ca4af333c0e809","opaqueData":""}]}' 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.003445Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13497999451699087314","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"20","size":3010802,"eTag":"f468f81db8f0055e012ef3ca41767f76","versionId":"","sequencer":"24E62366E27C3001","metadata":[],"tags":[]}},"eventId":"1713628708.019954.f468f81db8f0055e012ef3ca41767f76","opaqueData":""}]}' 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.022809Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14513780287258087763","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"21","size":3009895,"eTag":"d864fe4b83f2c226e6aabbae855a5483","versionId":"","sequencer":"24E6236651111402","metadata":[],"tags":[]}},"eventId":"1713628708.034869.d864fe4b83f2c226e6aabbae855a5483","opaqueData":""}]}' 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.027485Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17014508396727128221","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"23","size":3010537,"eTag":"61281c91877b9d33cccec105dd70aa74","versionId":"","sequencer":"24E62366BCE85E02","metadata":[],"tags":[]}},"eventId":"1713628708.039774.61281c91877b9d33cccec105dd70aa74","opaqueData":""}]}' 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/37 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/37 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/37 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lWlH6o5jURQEQv6Cic5mRA==', 'Content-Length': '3010878', 'Expect': '100-Continue'} 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.164 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:lWlH6o5jURQEQv6Cic5mRA== 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/37 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/HBYlTNm8ulhEaih//b4XNCddto= 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lWlH6o5jURQEQv6Cic5mRA==', 'Content-Length': '3010878', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/HBYlTNm8ulhEaih//b4XNCddto='} 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/38 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/38 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/38 2024-04-20T16:10:03.165 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'f3vTAyET4ZIwJVgN4MC73g==', 'Content-Length': '3011782', 'Expect': '100-Continue'} 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:f3vTAyET4ZIwJVgN4MC73g== 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/38 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:mwaN2uunEeWc2rtnrvAq1lQiS9I= 2024-04-20T16:10:03.166 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'f3vTAyET4ZIwJVgN4MC73g==', 'Content-Length': '3011782', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:mwaN2uunEeWc2rtnrvAq1lQiS9I='} 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/39 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/39 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/39 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l3tnY1s8QzOBP45kVhSdMg==', 'Content-Length': '3010056', 'Expect': '100-Continue'} 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:l3tnY1s8QzOBP45kVhSdMg== 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.167 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/39 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:b71bcHJoduOheqa09lCeNa022Bc= 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l3tnY1s8QzOBP45kVhSdMg==', 'Content-Length': '3010056', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:b71bcHJoduOheqa09lCeNa022Bc='} 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/40 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/40 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/40 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'mYGnFLLNzpnYdmLMRr6lnA==', 'Content-Length': '3010322', 'Expect': '100-Continue'} 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.168 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:mYGnFLLNzpnYdmLMRr6lnA== 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/40 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ok48/fwkCRH2wnWGzmPsLaJYwr8= 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'mYGnFLLNzpnYdmLMRr6lnA==', 'Content-Length': '3010322', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ok48/fwkCRH2wnWGzmPsLaJYwr8='} 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/41 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/41 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/41 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'PnItqDhPIis3U/oZSNLp8g==', 'Content-Length': '3010700', 'Expect': '100-Continue'} 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.169 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:PnItqDhPIis3U/oZSNLp8g== 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/41 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Pk9PF7J2lmJCNYbHLWSBP1LJFio= 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'PnItqDhPIis3U/oZSNLp8g==', 'Content-Length': '3010700', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Pk9PF7J2lmJCNYbHLWSBP1LJFio='} 2024-04-20T16:10:03.170 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dea71fd3bc35fdcb47ca4af333c0e809"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ba5775249e52a5c-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.038678Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8438953816189397881","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"22","size":3011758,"eTag":"bd9ceabd322dd8b8e7cd2af3b1bbf42a","versionId":"","sequencer":"24E6236694ED4E03","metadata":[],"tags":[]}},"eventId":"1713628708.055504.bd9ceabd322dd8b8e7cd2af3b1bbf42a","opaqueData":""}]}' 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.122137Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.925926257626801717","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"24","size":3011330,"eTag":"166083172e5cbd9c6c10d5bfe53b6a2d","versionId":"","sequencer":"24E62366CE762108","metadata":[],"tags":[]}},"eventId":"1713628708.136410.166083172e5cbd9c6c10d5bfe53b6a2d","opaqueData":""}]}' 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.121438Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8693856542917434214","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"25","size":3012033,"eTag":"6058717590ab8c051635880533625d81","versionId":"","sequencer":"24E6236634435708","metadata":[],"tags":[]}},"eventId":"1713628708.139936.6058717590ab8c051635880533625d81","opaqueData":""}]}' 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.906279Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9699874087643832644","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"19","size":3009477,"eTag":"4e0ca7c145633dbf96b480790524f643","versionId":"","sequencer":"24E6236678A1A002","metadata":[],"tags":[]}},"eventId":"1713628708.044081.4e0ca7c145633dbf96b480790524f643","opaqueData":""}]}' 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.171 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.873935Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13341218863965237301","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"17","size":3010090,"eTag":"eb4145ec27c25853ff69dad8dbe28e54","versionId":"","sequencer":"24E62366AD48A102","metadata":[],"tags":[]}},"eventId":"1713628708.044124.eb4145ec27c25853ff69dad8dbe28e54","opaqueData":""}]}' 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:27.906261Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15533824819609236920","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"18","size":3009133,"eTag":"95200555be069923d22b3157d4deb2c8","versionId":"","sequencer":"24E62366E488A002","metadata":[],"tags":[]}},"eventId":"1713628708.044075.95200555be069923d22b3157d4deb2c8","opaqueData":""}]}' 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/42 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/42 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/42 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'L3cTZDLya/+8hh77uZLKrA==', 'Content-Length': '3013207', 'Expect': '100-Continue'} 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.173 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:L3cTZDLya/+8hh77uZLKrA== 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/42 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:GPjtMUDpe2N4gJpVIiEhpF+nD8c= 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'L3cTZDLya/+8hh77uZLKrA==', 'Content-Length': '3013207', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:GPjtMUDpe2N4gJpVIiEhpF+nD8c='} 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/44 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/44 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/44 2024-04-20T16:10:03.174 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 's2H8kuAVf9fuIqsz1iejtw==', 'Content-Length': '3009962', 'Expect': '100-Continue'} 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:s2H8kuAVf9fuIqsz1iejtw== 2024-04-20T16:10:03.175 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/44 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:sJ6k2mh2nAa+GPUYfm7dzz8FKGs= 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 's2H8kuAVf9fuIqsz1iejtw==', 'Content-Length': '3009962', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:sJ6k2mh2nAa+GPUYfm7dzz8FKGs='} 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/45 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/45 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/45 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'nNnhnKqq9ioCRBXPFgh/Ww==', 'Content-Length': '3011973', 'Expect': '100-Continue'} 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.176 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:nNnhnKqq9ioCRBXPFgh/Ww== 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/45 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ha21wIX/aFNBSvCNOyItByl7lgY= 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'nNnhnKqq9ioCRBXPFgh/Ww==', 'Content-Length': '3011973', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ha21wIX/aFNBSvCNOyItByl7lgY='} 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/43 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/43 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.177 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/43 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TojBOI0bYAvu2g/KJN6yRg==', 'Content-Length': '3011046', 'Expect': '100-Continue'} 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:TojBOI0bYAvu2g/KJN6yRg== 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/43 2024-04-20T16:10:03.178 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:QFGcbhp4Xc30LB8sBKE6IQ8iCFI= 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TojBOI0bYAvu2g/KJN6yRg==', 'Content-Length': '3011046', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:QFGcbhp4Xc30LB8sBKE6IQ8iCFI='} 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/46 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/46 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/46 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'XgZMjnJn+1uVdPzetSNN8Q==', 'Content-Length': '3011653', 'Expect': '100-Continue'} 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.179 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:XgZMjnJn+1uVdPzetSNN8Q== 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/46 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:IP1p66pUvmUYLYyju7U6dvUzus0= 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'XgZMjnJn+1uVdPzetSNN8Q==', 'Content-Length': '3011653', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:IP1p66pUvmUYLYyju7U6dvUzus0='} 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"eb4145ec27c25853ff69dad8dbe28e54"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b92587df39637035-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"95200555be069923d22b3157d4deb2c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7933b5f27f475b8-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"4e0ca7c145633dbf96b480790524f643"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000869ce0173006e144-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.160855Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.11779530006374881640","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"26","size":3011903,"eTag":"12f853aa57e72be5a3dcca70d8d7c17a","versionId":"","sequencer":"24E6236655DD480A","metadata":[],"tags":[]}},"eventId":"1713628708.172547.12f853aa57e72be5a3dcca70d8d7c17a","opaqueData":""}]}' 2024-04-20T16:10:03.180 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.212247Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.183508202278497855","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"28","size":3009833,"eTag":"308114d39d4f1f15bd9389d63dc26463","versionId":"","sequencer":"24E623669047730D","metadata":[],"tags":[]}},"eventId":"1713628708.225658.308114d39d4f1f15bd9389d63dc26463","opaqueData":""}]}' 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.222719Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.5701922769181957600","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"27","size":3011781,"eTag":"943bc47f6783fc2fd61e56ade26e5061","versionId":"","sequencer":"24E62366F20D350E","metadata":[],"tags":[]}},"eventId":"1713628708.238358.943bc47f6783fc2fd61e56ade26e5061","opaqueData":""}]}' 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.027485Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17014508396727128221","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"23","size":3010537,"eTag":"61281c91877b9d33cccec105dd70aa74","versionId":"","sequencer":"24E62366785C9908","metadata":[],"tags":[]}},"eventId":"1713628708.144268.61281c91877b9d33cccec105dd70aa74","opaqueData":""}]}' 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.003445Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13497999451699087314","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"20","size":3010802,"eTag":"f468f81db8f0055e012ef3ca41767f76","versionId":"","sequencer":"24E623669E439908","metadata":[],"tags":[]}},"eventId":"1713628708.144262.f468f81db8f0055e012ef3ca41767f76","opaqueData":""}]}' 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.022809Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14513780287258087763","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"21","size":3009895,"eTag":"d864fe4b83f2c226e6aabbae855a5483","versionId":"","sequencer":"24E62366A05A9908","metadata":[],"tags":[]}},"eventId":"1713628708.144267.d864fe4b83f2c226e6aabbae855a5483","opaqueData":""}]}' 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/48 2024-04-20T16:10:03.181 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/48 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/48 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e7NSxj+8chNaheggM+BZUg==', 'Content-Length': '3010960', 'Expect': '100-Continue'} 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:e7NSxj+8chNaheggM+BZUg== 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.182 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/48 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:0Jn6QlX181yGvpNHZnsvOiWC18I= 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e7NSxj+8chNaheggM+BZUg==', 'Content-Length': '3010960', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:0Jn6QlX181yGvpNHZnsvOiWC18I='} 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/47 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/47 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/47 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VnzruRw8GKG4RSkQKhKQMw==', 'Content-Length': '3008151', 'Expect': '100-Continue'} 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.183 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:VnzruRw8GKG4RSkQKhKQMw== 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/47 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:fTjHjAD6o33QmmhBj1M5ss78Yeo= 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VnzruRw8GKG4RSkQKhKQMw==', 'Content-Length': '3008151', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:fTjHjAD6o33QmmhBj1M5ss78Yeo='} 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/49 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/49 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/49 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebJZjeyKzI2wNMXRXEMYZg==', 'Content-Length': '3011063', 'Expect': '100-Continue'} 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.184 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:ebJZjeyKzI2wNMXRXEMYZg== 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/49 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:rQiW8jP2oXoa551BKrhz8lji3os= 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebJZjeyKzI2wNMXRXEMYZg==', 'Content-Length': '3011063', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:rQiW8jP2oXoa551BKrhz8lji3os='} 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/50 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/50 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.185 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/50 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'nYv6nedRaMoqIvcEkrUwFA==', 'Content-Length': '3011009', 'Expect': '100-Continue'} 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:nYv6nedRaMoqIvcEkrUwFA== 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/50 2024-04-20T16:10:03.186 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:bNQoypvkxGaX9B9i1wDN14Q1n+I= 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'nYv6nedRaMoqIvcEkrUwFA==', 'Content-Length': '3011009', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:bNQoypvkxGaX9B9i1wDN14Q1n+I='} 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/51 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/51 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/51 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Fihbo1ygBULSYzkPj6O8Hw==', 'Content-Length': '3011022', 'Expect': '100-Continue'} 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.187 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:Fihbo1ygBULSYzkPj6O8Hw== 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/51 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:qvSqIRNUKhO6pe0HgPXmrEIzeiU= 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Fihbo1ygBULSYzkPj6O8Hw==', 'Content-Length': '3011022', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:qvSqIRNUKhO6pe0HgPXmrEIzeiU='} 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"f468f81db8f0055e012ef3ca41767f76"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb5286fc60f107d2-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d864fe4b83f2c226e6aabbae855a5483"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c96b4e462215d153-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"61281c91877b9d33cccec105dd70aa74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec1fad3d76d2509d-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.188 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.122137Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.925926257626801717","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"24","size":3011330,"eTag":"166083172e5cbd9c6c10d5bfe53b6a2d","versionId":"","sequencer":"24E62366295D920E","metadata":[],"tags":[]}},"eventId":"1713628708.244473.166083172e5cbd9c6c10d5bfe53b6a2d","opaqueData":""}]}' 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.280719Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12464291464894905787","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"30","size":3012542,"eTag":"266ead75809220d818f5286c0b03c33e","versionId":"","sequencer":"24E62366DE7C9C11","metadata":[],"tags":[]}},"eventId":"1713628708.295468.266ead75809220d818f5286c0b03c33e","opaqueData":""}]}' 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/52 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.038678Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8438953816189397881","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"22","size":3011758,"eTag":"bd9ceabd322dd8b8e7cd2af3b1bbf42a","versionId":"","sequencer":"24E62366795B920E","metadata":[],"tags":[]}},"eventId":"1713628708.244472.bd9ceabd322dd8b8e7cd2af3b1bbf42a","opaqueData":""}]}' 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.283515Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7202946691049244106","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"29","size":3011722,"eTag":"0915fab826732a73d1de92fdcca49586","versionId":"","sequencer":"24E623660953B111","metadata":[],"tags":[]}},"eventId":"1713628708.296833.0915fab826732a73d1de92fdcca49586","opaqueData":""}]}' 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/52 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.121438Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8693856542917434214","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"25","size":3012033,"eTag":"6058717590ab8c051635880533625d81","versionId":"","sequencer":"24E6236680C8920E","metadata":[],"tags":[]}},"eventId":"1713628708.244500.6058717590ab8c051635880533625d81","opaqueData":""}]}' 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/52 2024-04-20T16:10:03.189 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BovqqYnWW+dy7ptDJLNNaQ==', 'Content-Length': '3010255', 'Expect': '100-Continue'} 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:BovqqYnWW+dy7ptDJLNNaQ== 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/52 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:mLMgV6+a70JP2VnLg5eySz4fa34= 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BovqqYnWW+dy7ptDJLNNaQ==', 'Content-Length': '3010255', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:mLMgV6+a70JP2VnLg5eySz4fa34='} 2024-04-20T16:10:03.190 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/54 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/54 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/54 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CKydn/zN1T8CuoS7Vf/QyQ==', 'Content-Length': '3011708', 'Expect': '100-Continue'} 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:CKydn/zN1T8CuoS7Vf/QyQ== 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.191 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/54 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DIA231GJFjczR0gfd4CoafFXMy8= 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CKydn/zN1T8CuoS7Vf/QyQ==', 'Content-Length': '3011708', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DIA231GJFjczR0gfd4CoafFXMy8='} 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/53 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/53 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/53 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'tltSCzlxoDm2injI45FfdQ==', 'Content-Length': '3010204', 'Expect': '100-Continue'} 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.192 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:tltSCzlxoDm2injI45FfdQ== 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/53 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:IdPi3SRFggTMcTB6cn+mzuTO2+4= 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'tltSCzlxoDm2injI45FfdQ==', 'Content-Length': '3010204', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:IdPi3SRFggTMcTB6cn+mzuTO2+4='} 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/55 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/55 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/55 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.193 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '225+hy71E2lCStV5m1f8/A==', 'Content-Length': '3010361', 'Expect': '100-Continue'} 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:225+hy71E2lCStV5m1f8/A== 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/55 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:mbhoBaHUyrfitytdsiQ9WsgfHk4= 2024-04-20T16:10:03.194 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '225+hy71E2lCStV5m1f8/A==', 'Content-Length': '3010361', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:mbhoBaHUyrfitytdsiQ9WsgfHk4='} 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"bd9ceabd322dd8b8e7cd2af3b1bbf42a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000751d2fc8c9e56b79-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6058717590ab8c051635880533625d81"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078a6c876b8cc9b66-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.388394Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1657567704726184209","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"32","size":3011609,"eTag":"904a20ac62763beab3cfcb01c4861896","versionId":"","sequencer":"24E62366DB4FF417","metadata":[],"tags":[]}},"eventId":"1713628708.401887.904a20ac62763beab3cfcb01c4861896","opaqueData":""}]}' 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.416596Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7304926041507317130","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"31","size":3009651,"eTag":"a9d26713272130b9663655284e89318c","versionId":"","sequencer":"24E62366D6CED119","metadata":[],"tags":[]}},"eventId":"1713628708.433180.a9d26713272130b9663655284e89318c","opaqueData":""}]}' 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.419535Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4905550373426557253","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"34","size":3010350,"eTag":"6022a6efddc8bf045ad8a9a50a1bba4e","versionId":"","sequencer":"24E62366FD7DD419","metadata":[],"tags":[]}},"eventId":"1713628708.433356.6022a6efddc8bf045ad8a9a50a1bba4e","opaqueData":""}]}' 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.222719Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.5701922769181957600","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"27","size":3011781,"eTag":"943bc47f6783fc2fd61e56ade26e5061","versionId":"","sequencer":"24E62366F1038B14","metadata":[],"tags":[]}},"eventId":"1713628708.344654.943bc47f6783fc2fd61e56ade26e5061","opaqueData":""}]}' 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.212247Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.183508202278497855","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"28","size":3009833,"eTag":"308114d39d4f1f15bd9389d63dc26463","versionId":"","sequencer":"24E6236659148B14","metadata":[],"tags":[]}},"eventId":"1713628708.344659.308114d39d4f1f15bd9389d63dc26463","opaqueData":""}]}' 2024-04-20T16:10:03.195 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.160855Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.11779530006374881640","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"26","size":3011903,"eTag":"12f853aa57e72be5a3dcca70d8d7c17a","versionId":"","sequencer":"24E623667F078B14","metadata":[],"tags":[]}},"eventId":"1713628708.344655.12f853aa57e72be5a3dcca70d8d7c17a","opaqueData":""}]}' 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"166083172e5cbd9c6c10d5bfe53b6a2d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000cd98d068a0d6635-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/56 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/56 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/56 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'n4Bsj/iH4ZA+sqlh70DEeA==', 'Content-Length': '3010584', 'Expect': '100-Continue'} 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.196 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:n4Bsj/iH4ZA+sqlh70DEeA== 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/56 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PvGxRKYOi+w5sqQwcXYpEAXk6B4= 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'n4Bsj/iH4ZA+sqlh70DEeA==', 'Content-Length': '3010584', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PvGxRKYOi+w5sqQwcXYpEAXk6B4='} 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/57 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/57 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/57 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7pblRBPVvi7Wwbr+jDfiwQ==', 'Content-Length': '3011645', 'Expect': '100-Continue'} 2024-04-20T16:10:03.197 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:7pblRBPVvi7Wwbr+jDfiwQ== 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/57 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:K5021JHmoK0DUHDNCCChEGXeZ4A= 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7pblRBPVvi7Wwbr+jDfiwQ==', 'Content-Length': '3011645', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:K5021JHmoK0DUHDNCCChEGXeZ4A='} 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/59 2024-04-20T16:10:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/59 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/59 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HWwtO6QBniP3WonMtkBiiw==', 'Content-Length': '3011792', 'Expect': '100-Continue'} 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:HWwtO6QBniP3WonMtkBiiw== 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/59 2024-04-20T16:10:03.199 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:p/AHmZ/36fKu6pwvqYI91PmOs2o= 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HWwtO6QBniP3WonMtkBiiw==', 'Content-Length': '3011792', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:p/AHmZ/36fKu6pwvqYI91PmOs2o='} 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/58 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/58 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/58 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'UcDH07qfrpZekjd/JfZRbQ==', 'Content-Length': '3009467', 'Expect': '100-Continue'} 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.200 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:UcDH07qfrpZekjd/JfZRbQ== 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/58 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:0sSq+gCU+ZPThz7FFM2VF7UGzdc= 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'UcDH07qfrpZekjd/JfZRbQ==', 'Content-Length': '3009467', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:0sSq+gCU+ZPThz7FFM2VF7UGzdc='} 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"12f853aa57e72be5a3dcca70d8d7c17a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3794c2153079968-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.283515Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7202946691049244106","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"29","size":3011722,"eTag":"0915fab826732a73d1de92fdcca49586","versionId":"","sequencer":"24E623668AC7831A","metadata":[],"tags":[]}},"eventId":"1713628708.444843.0915fab826732a73d1de92fdcca49586","opaqueData":""}]}' 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.280719Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12464291464894905787","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"30","size":3012542,"eTag":"266ead75809220d818f5286c0b03c33e","versionId":"","sequencer":"24E623667BC7831A","metadata":[],"tags":[]}},"eventId":"1713628708.444843.266ead75809220d818f5286c0b03c33e","opaqueData":""}]}' 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"943bc47f6783fc2fd61e56ade26e5061"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f214c8c69cd5de0-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.201 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.439333Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14414658098105397532","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"33","size":3010798,"eTag":"c4cce01f8d44323faa0985339ec7ada6","versionId":"","sequencer":"24E62366AFA13A1B","metadata":[],"tags":[]}},"eventId":"1713628708.456827.c4cce01f8d44323faa0985339ec7ada6","opaqueData":""}]}' 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.460283Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1893268354287502040","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"35","size":3010488,"eTag":"bf2e05d77cc2a212a379f789e6ea312b","versionId":"","sequencer":"24E62366A928221C","metadata":[],"tags":[]}},"eventId":"1713628708.472000.bf2e05d77cc2a212a379f789e6ea312b","opaqueData":""}]}' 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.497935Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7728169859866015689","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"36","size":3008755,"eTag":"0c90f4f1afb6e813d996d08b5db544e3","versionId":"","sequencer":"24E62366907F671E","metadata":[],"tags":[]}},"eventId":"1713628708.510099.0c90f4f1afb6e813d996d08b5db544e3","opaqueData":""}]}' 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.516810Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.18076408548287478107","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"38","size":3011782,"eTag":"7f7bd3032113e1923025580de0c0bbde","versionId":"","sequencer":"24E6236657B9971F","metadata":[],"tags":[]}},"eventId":"1713628708.530037.7f7bd3032113e1923025580de0c0bbde","opaqueData":""}]}' 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"308114d39d4f1f15bd9389d63dc26463"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000028bf3bcda2e2e3f-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/60 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/60 2024-04-20T16:10:03.202 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/60 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qAAumdU/iSsIXecloFqKnQ==', 'Content-Length': '3009879', 'Expect': '100-Continue'} 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:qAAumdU/iSsIXecloFqKnQ== 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/60 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.203 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:TGk4p2oh24xgxxhN7Q8y8b0UW78= 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qAAumdU/iSsIXecloFqKnQ==', 'Content-Length': '3009879', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:TGk4p2oh24xgxxhN7Q8y8b0UW78='} 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/61 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/61 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/61 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hXX6PJPm5egRa45Y+FtluA==', 'Content-Length': '3009352', 'Expect': '100-Continue'} 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:hXX6PJPm5egRa45Y+FtluA== 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/61 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DOeMB8YE3y4qNSwfyPEWgG5YtEU= 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hXX6PJPm5egRa45Y+FtluA==', 'Content-Length': '3009352', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DOeMB8YE3y4qNSwfyPEWgG5YtEU='} 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/62 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/62 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/62 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '+tOjyVLCYt6tGsgol/u37Q==', 'Content-Length': '3008324', 'Expect': '100-Continue'} 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.205 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:+tOjyVLCYt6tGsgol/u37Q== 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/62 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KLXapPJSZstJhmX/rAafAx8DZWc= 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '+tOjyVLCYt6tGsgol/u37Q==', 'Content-Length': '3008324', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KLXapPJSZstJhmX/rAafAx8DZWc='} 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/64 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/64 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.206 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/64 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AZp74Zx89LLeU+qLQ8VaQA==', 'Content-Length': '3007840', 'Expect': '100-Continue'} 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:AZp74Zx89LLeU+qLQ8VaQA== 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/64 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.207 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:r2hzT1VR/90R3WNuzxPf8wgy13A= 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AZp74Zx89LLeU+qLQ8VaQA==', 'Content-Length': '3007840', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:r2hzT1VR/90R3WNuzxPf8wgy13A='} 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/63 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/63 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/63 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'b5fdStSgVFIRr9wTKI31Ow==', 'Content-Length': '3013115', 'Expect': '100-Continue'} 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:b5fdStSgVFIRr9wTKI31Ow== 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/63 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:gO0o1b2bBkMI/rRUi9tXaYjuA9M= 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'b5fdStSgVFIRr9wTKI31Ow==', 'Content-Length': '3013115', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:gO0o1b2bBkMI/rRUi9tXaYjuA9M='} 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/65 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/65 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/65 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lN9xE+mRkyX3S/bSQ5My/A==', 'Content-Length': '3009881', 'Expect': '100-Continue'} 2024-04-20T16:10:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:lN9xE+mRkyX3S/bSQ5My/A== 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/65 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:ZX2aBK4x/8oZYXnE5i7exu9RVB8= 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lN9xE+mRkyX3S/bSQ5My/A==', 'Content-Length': '3009881', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:ZX2aBK4x/8oZYXnE5i7exu9RVB8='} 2024-04-20T16:10:03.210 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"0915fab826732a73d1de92fdcca49586"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063f601da4810c5ca-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"266ead75809220d818f5286c0b03c33e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acfa0f0c75fe7dbb-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.543957Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.2451127753215654967","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"37","size":3010878,"eTag":"956947ea8e6351140442fe8289ce6644","versionId":"","sequencer":"24E62366D2A60121","metadata":[],"tags":[]}},"eventId":"1713628708.553756.956947ea8e6351140442fe8289ce6644","opaqueData":""}]}' 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.582053Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16244602140156202536","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"39","size":3010056,"eTag":"977b67635b3c4333813f8e6456149d32","versionId":"","sequencer":"24E6236601386123","metadata":[],"tags":[]}},"eventId":"1713628708.593573.977b67635b3c4333813f8e6456149d32","opaqueData":""}]}' 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.612558Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1741642334581290933","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"40","size":3010322,"eTag":"9981a714b2cdce99d87662cc46bea59c","versionId":"","sequencer":"24E62366FA1A0A25","metadata":[],"tags":[]}},"eventId":"1713628708.621419.9981a714b2cdce99d87662cc46bea59c","opaqueData":""}]}' 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.416596Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7304926041507317130","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"31","size":3009651,"eTag":"a9d26713272130b9663655284e89318c","versionId":"","sequencer":"24E6236665587C20","metadata":[],"tags":[]}},"eventId":"1713628708.545020.a9d26713272130b9663655284e89318c","opaqueData":""}]}' 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.388394Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1657567704726184209","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"32","size":3011609,"eTag":"904a20ac62763beab3cfcb01c4861896","versionId":"","sequencer":"24E62366F9287C20","metadata":[],"tags":[]}},"eventId":"1713628708.545007.904a20ac62763beab3cfcb01c4861896","opaqueData":""}]}' 2024-04-20T16:10:03.211 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.419535Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4905550373426557253","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"34","size":3010350,"eTag":"6022a6efddc8bf045ad8a9a50a1bba4e","versionId":"","sequencer":"24E623665C6E7C20","metadata":[],"tags":[]}},"eventId":"1713628708.545025.6022a6efddc8bf045ad8a9a50a1bba4e","opaqueData":""}]}' 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/66 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/66 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/66 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '9L9IGiwjOO76GUVVj/sCdg==', 'Content-Length': '3010129', 'Expect': '100-Continue'} 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.212 INFO:teuthology.orchestra.run.smithi071.stderr:9L9IGiwjOO76GUVVj/sCdg== 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/66 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:onnt4gXgTDaKGxGRuv2WZPYmJyE= 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '9L9IGiwjOO76GUVVj/sCdg==', 'Content-Length': '3010129', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:onnt4gXgTDaKGxGRuv2WZPYmJyE='} 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/68 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/68 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/68 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '16EKVvvEf+s5ZSmh+miGdg==', 'Content-Length': '3010165', 'Expect': '100-Continue'} 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.213 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:16EKVvvEf+s5ZSmh+miGdg== 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/68 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/BOr/zJoUDjkNZfSwsqbEZnkrlc= 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '16EKVvvEf+s5ZSmh+miGdg==', 'Content-Length': '3010165', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/BOr/zJoUDjkNZfSwsqbEZnkrlc='} 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/69 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/69 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/69 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Wyy2ZXjxw+mBWaLqTYQsQw==', 'Content-Length': '3008237', 'Expect': '100-Continue'} 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:Wyy2ZXjxw+mBWaLqTYQsQw== 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/69 2024-04-20T16:10:03.215 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:+18SdEyorSrLmXgaqJU8tv7Jw+Q= 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Wyy2ZXjxw+mBWaLqTYQsQw==', 'Content-Length': '3008237', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:+18SdEyorSrLmXgaqJU8tv7Jw+Q='} 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/67 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/67 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/67 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6W2iTuDvfaHvRT3D1vyTFg==', 'Content-Length': '3010401', 'Expect': '100-Continue'} 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.216 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:6W2iTuDvfaHvRT3D1vyTFg== 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/67 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:gRiaOdZOP5x7XhDDKrfYiUAdEc8= 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6W2iTuDvfaHvRT3D1vyTFg==', 'Content-Length': '3010401', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:gRiaOdZOP5x7XhDDKrfYiUAdEc8='} 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a9d26713272130b9663655284e89318c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065604f88168b458a-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"904a20ac62763beab3cfcb01c4861896"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001700dd0b975d2111-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.439333Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14414658098105397532","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"33","size":3010798,"eTag":"c4cce01f8d44323faa0985339ec7ada6","versionId":"","sequencer":"24E6236619047526","metadata":[],"tags":[]}},"eventId":"1713628708.645202.c4cce01f8d44323faa0985339ec7ada6","opaqueData":""}]}' 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.497935Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7728169859866015689","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"36","size":3008755,"eTag":"0c90f4f1afb6e813d996d08b5db544e3","versionId":"","sequencer":"24E62366F4E77526","metadata":[],"tags":[]}},"eventId":"1713628708.645261.0c90f4f1afb6e813d996d08b5db544e3","opaqueData":""}]}' 2024-04-20T16:10:03.217 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.460283Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1893268354287502040","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"35","size":3010488,"eTag":"bf2e05d77cc2a212a379f789e6ea312b","versionId":"","sequencer":"24E6236686807526","metadata":[],"tags":[]}},"eventId":"1713628708.645234.bf2e05d77cc2a212a379f789e6ea312b","opaqueData":""}]}' 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.516810Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.18076408548287478107","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"38","size":3011782,"eTag":"7f7bd3032113e1923025580de0c0bbde","versionId":"","sequencer":"24E6236600FD7526","metadata":[],"tags":[]}},"eventId":"1713628708.645266.7f7bd3032113e1923025580de0c0bbde","opaqueData":""}]}' 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.649411Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6886667351242998518","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"41","size":3010700,"eTag":"3e722da8384f222b3753fa1948d2e9f2","versionId":"","sequencer":"24E62366B60B6927","metadata":[],"tags":[]}},"eventId":"1713628708.661195.3e722da8384f222b3753fa1948d2e9f2","opaqueData":""}]}' 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.700168Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4847786624915674650","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"45","size":3011973,"eTag":"9cd9e19caaaaf62a024415cf16087f5b","versionId":"","sequencer":"24E623665A2D802A","metadata":[],"tags":[]}},"eventId":"1713628708.713043.9cd9e19caaaaf62a024415cf16087f5b","opaqueData":""}]}' 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.721369Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17641235987686756454","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"44","size":3009962,"eTag":"b361fc92e0157fd7ee22ab33d627a3b7","versionId":"","sequencer":"24E62366A324B42B","metadata":[],"tags":[]}},"eventId":"1713628708.733226.b361fc92e0157fd7ee22ab33d627a3b7","opaqueData":""}]}' 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6022a6efddc8bf045ad8a9a50a1bba4e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044140412ebdff145-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/72 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/72 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/72 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uFnQMIT8/U9TLx6WCsJvyw==', 'Content-Length': '3012871', 'Expect': '100-Continue'} 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:uFnQMIT8/U9TLx6WCsJvyw== 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.219 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/72 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DOkou5MW3qGiw6jj8kcJaP2fVh4= 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uFnQMIT8/U9TLx6WCsJvyw==', 'Content-Length': '3012871', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DOkou5MW3qGiw6jj8kcJaP2fVh4='} 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/70 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/70 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/70 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'D1fcSHMzQw9xCNbkTYny0Q==', 'Content-Length': '3011693', 'Expect': '100-Continue'} 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.220 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:D1fcSHMzQw9xCNbkTYny0Q== 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/70 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Zp4LbvI9jTwWAoQ0Y068zcxK/dc= 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'D1fcSHMzQw9xCNbkTYny0Q==', 'Content-Length': '3011693', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Zp4LbvI9jTwWAoQ0Y068zcxK/dc='} 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/71 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/71 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/71 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'oQs1f1rKw/rHoVxVLZkDLA==', 'Content-Length': '3008799', 'Expect': '100-Continue'} 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.221 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:oQs1f1rKw/rHoVxVLZkDLA== 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/71 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:ZxQ5IbB9jbqWKfWD2L5lh2UUw/U= 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'oQs1f1rKw/rHoVxVLZkDLA==', 'Content-Length': '3008799', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:ZxQ5IbB9jbqWKfWD2L5lh2UUw/U='} 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/74 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/74 2024-04-20T16:10:03.222 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/74 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BOzUaQ7l6Pg6WIdocTqRug==', 'Content-Length': '3011849', 'Expect': '100-Continue'} 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:BOzUaQ7l6Pg6WIdocTqRug== 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/74 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.223 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rh39Uql+dZOklgDfPuwga9tZ/fg= 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BOzUaQ7l6Pg6WIdocTqRug==', 'Content-Length': '3011849', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rh39Uql+dZOklgDfPuwga9tZ/fg='} 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/73 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/73 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/73 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TSDcEJAPkhrRGuBSCMjL8w==', 'Content-Length': '3011501', 'Expect': '100-Continue'} 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.224 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:TSDcEJAPkhrRGuBSCMjL8w== 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/73 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:J0uKJ0UujI/kWcWWlFe2imclGck= 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'TSDcEJAPkhrRGuBSCMjL8w==', 'Content-Length': '3011501', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:J0uKJ0UujI/kWcWWlFe2imclGck='} 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/75 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/75 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/75 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.225 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'bR2Ggr6wCeWToIQDgJk4AA==', 'Content-Length': '3008117', 'Expect': '100-Continue'} 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:bR2Ggr6wCeWToIQDgJk4AA== 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/75 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:iLb845pV26M5v9RukIDp35yhWUg= 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'bR2Ggr6wCeWToIQDgJk4AA==', 'Content-Length': '3008117', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:iLb845pV26M5v9RukIDp35yhWUg='} 2024-04-20T16:10:03.226 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/76 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/76 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/76 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue'} 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.227 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.543957Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.2451127753215654967","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"37","size":3010878,"eTag":"956947ea8e6351140442fe8289ce6644","versionId":"","sequencer":"24E6236624E86D2C","metadata":[],"tags":[]}},"eventId":"1713628708.745400.956947ea8e6351140442fe8289ce6644","opaqueData":""}]}' 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.612558Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1741642334581290933","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"40","size":3010322,"eTag":"9981a714b2cdce99d87662cc46bea59c","versionId":"","sequencer":"24E6236647566E2C","metadata":[],"tags":[]}},"eventId":"1713628708.745428.9981a714b2cdce99d87662cc46bea59c","opaqueData":""}]}' 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.582053Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16244602140156202536","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"39","size":3010056,"eTag":"977b67635b3c4333813f8e6456149d32","versionId":"","sequencer":"24E623669D566E2C","metadata":[],"tags":[]}},"eventId":"1713628708.745428.977b67635b3c4333813f8e6456149d32","opaqueData":""}]}' 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7f7bd3032113e1923025580de0c0bbde"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fadc4dcd3f82795b-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"c4cce01f8d44323faa0985339ec7ada6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c80b272b0847b11c-006623e623-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"bf2e05d77cc2a212a379f789e6ea312b"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a463d8989cb82d8-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"0c90f4f1afb6e813d996d08b5db544e3"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b3ff98abd31b7c9-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.735017Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15339365163889935341","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"42","size":3013207,"eTag":"2f77136432f26bffbc861efbb992caac","versionId":"","sequencer":"24E62366BBD9B92C","metadata":[],"tags":[]}},"eventId":"1713628708.750377.2f77136432f26bffbc861efbb992caac","opaqueData":""}]}' 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.734403Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17403901371603164130","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"43","size":3011046,"eTag":"4e88c1388d1b600beeda0fca24deb246","versionId":"","sequencer":"24E623666022BF2C","metadata":[],"tags":[]}},"eventId":"1713628708.750723.4e88c1388d1b600beeda0fca24deb246","opaqueData":""}]}' 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.753080Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15931644201860767922","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"46","size":3011653,"eTag":"5e064c8e7267fb5b9574fcdeb5234df1","versionId":"","sequencer":"24E62366DE32AF2D","metadata":[],"tags":[]}},"eventId":"1713628708.766456.5e064c8e7267fb5b9574fcdeb5234df1","opaqueData":""}]}' 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.778684Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.18340924620923969834","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"48","size":3010960,"eTag":"7bb352c63fbc72135a85e82033e05952","versionId":"","sequencer":"24E62366BF8FF82E","metadata":[],"tags":[]}},"eventId":"1713628708.788041.7bb352c63fbc72135a85e82033e05952","opaqueData":""}]}' 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.229 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.792029Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14732378859543301633","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"47","size":3008151,"eTag":"567cebb91c3c18a1b84529102a129033","versionId":"","sequencer":"24E6236642D4F22F","metadata":[],"tags":[]}},"eventId":"1713628708.804443.567cebb91c3c18a1b84529102a129033","opaqueData":""}]}' 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.828082Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6402583333206743309","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"50","size":3011009,"eTag":"9d8bfa9de75168ca2a22f70492b53014","versionId":"","sequencer":"24E6236684742832","metadata":[],"tags":[]}},"eventId":"1713628708.841512.9d8bfa9de75168ca2a22f70492b53014","opaqueData":""}]}' 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/80 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/80 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/80 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue'} 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.230 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/78 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/78 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/78 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'F1RzizJGhkmvPeUxXt9ZCA==', 'Content-Length': '3011218', 'Expect': '100-Continue'} 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.231 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:F1RzizJGhkmvPeUxXt9ZCA== 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/78 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:9tkqL9cSkKF8pFjTjRAcap+f2L8= 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'F1RzizJGhkmvPeUxXt9ZCA==', 'Content-Length': '3011218', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:9tkqL9cSkKF8pFjTjRAcap+f2L8='} 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/81 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/81 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/81 2024-04-20T16:10:03.232 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Cv4wv/i0fIU7AgehwoEeew==', 'Content-Length': '3008315', 'Expect': '100-Continue'} 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:Cv4wv/i0fIU7AgehwoEeew== 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/81 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:fDJTVhNJSR47ED4yrM2D9PpSfTA= 2024-04-20T16:10:03.233 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Cv4wv/i0fIU7AgehwoEeew==', 'Content-Length': '3008315', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:fDJTVhNJSR47ED4yrM2D9PpSfTA='} 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/77 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/77 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/77 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7RXyph/Ia7jb6zizlqaBFQ==', 'Content-Length': '3011971', 'Expect': '100-Continue'} 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:7RXyph/Ia7jb6zizlqaBFQ== 2024-04-20T16:10:03.234 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/77 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DdpgswXNKvzi6f3fLPXA5PD2xko= 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '7RXyph/Ia7jb6zizlqaBFQ==', 'Content-Length': '3011971', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DdpgswXNKvzi6f3fLPXA5PD2xko='} 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/79 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/79 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/79 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'b4sW8HazQ+UdgZFUFsnUFw==', 'Content-Length': '3011496', 'Expect': '100-Continue'} 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.235 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:b4sW8HazQ+UdgZFUFsnUFw== 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/79 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:rWJwtDRDuV+lD7GgR9La2w8bwEM= 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'b4sW8HazQ+UdgZFUFsnUFw==', 'Content-Length': '3011496', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:rWJwtDRDuV+lD7GgR9La2w8bwEM='} 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/82 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/82 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/82 2024-04-20T16:10:03.236 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue'} 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.237 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/83 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/83 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/83 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue'} 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.238 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/85 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/85 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/85 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue'} 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.239 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/84 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/84 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.240 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/84 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue'} 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.241 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"956947ea8e6351140442fe8289ce6644"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000220427b475853c37-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"977b67635b3c4333813f8e6456149d32"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1706b94318f8e28-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/87 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/87 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/87 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue'} 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.242 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"3e722da8384f222b3753fa1948d2e9f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f925b78051e82f6-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"b361fc92e0157fd7ee22ab33d627a3b7"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4d242a9dce81066-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"9cd9e19caaaaf62a024415cf16087f5b"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004346cc402cfd161a-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"9981a714b2cdce99d87662cc46bea59c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000182b8e79c67ff7b5-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.844812Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15375269135918325354","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"49","size":3011063,"eTag":"79b2598dec8acc8db034c5d15c431866","versionId":"","sequencer":"24E6236696371333","metadata":[],"tags":[]}},"eventId":"1713628708.856897.79b2598dec8acc8db034c5d15c431866","opaqueData":""}]}' 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.874236Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4174137785450450970","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"51","size":3011022,"eTag":"16285ba35ca00542d263390f8fa3bc1f","versionId":"","sequencer":"24E623664C352735","metadata":[],"tags":[]}},"eventId":"1713628708.891761.16285ba35ca00542d263390f8fa3bc1f","opaqueData":""}]}' 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.882616Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14020226644781210991","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"54","size":3011708,"eTag":"08ac9d9ffccdd53f02ba84bb55ffd0c9","versionId":"","sequencer":"24E623668E374B35","metadata":[],"tags":[]}},"eventId":"1713628708.894121.08ac9d9ffccdd53f02ba84bb55ffd0c9","opaqueData":""}]}' 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.882872Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17258475958805296835","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"52","size":3010255,"eTag":"068beaa989d65be772ee9b4324b34d69","versionId":"","sequencer":"24E6236656989B35","metadata":[],"tags":[]}},"eventId":"1713628708.899389.068beaa989d65be772ee9b4324b34d69","opaqueData":""}]}' 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.888118Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8161191292885337777","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"53","size":3010204,"eTag":"b65b520b3971a039b68a78c8e3915f75","versionId":"","sequencer":"24E62366E600D035","metadata":[],"tags":[]}},"eventId":"1713628708.902824.b65b520b3971a039b68a78c8e3915f75","opaqueData":""}]}' 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.903348Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12000835659748803139","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"55","size":3010361,"eTag":"db6e7e872ef51369424ad5799b57fcfc","versionId":"","sequencer":"24E623669BE66436","metadata":[],"tags":[]}},"eventId":"1713628708.912582.db6e7e872ef51369424ad5799b57fcfc","opaqueData":""}]}' 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.244 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.924255Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7809800318522890568","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"56","size":3010584,"eTag":"9f806c8ff887e1903eb2a961ef40c478","versionId":"","sequencer":"24E62366571BF337","metadata":[],"tags":[]}},"eventId":"1713628708.938679.9f806c8ff887e1903eb2a961ef40c478","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.933007Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7645736487297238199","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"57","size":3011645,"eTag":"ee96e54413d5be2ed6c1bafe8c37e2c1","versionId":"","sequencer":"24E6236615A44338","metadata":[],"tags":[]}},"eventId":"1713628708.943957.ee96e54413d5be2ed6c1bafe8c37e2c1","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.700168Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4847786624915674650","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"45","size":3011973,"eTag":"9cd9e19caaaaf62a024415cf16087f5b","versionId":"","sequencer":"24E62366094A6732","metadata":[],"tags":[]}},"eventId":"1713628708.845629.9cd9e19caaaaf62a024415cf16087f5b","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.721369Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17641235987686756454","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"44","size":3009962,"eTag":"b361fc92e0157fd7ee22ab33d627a3b7","versionId":"","sequencer":"24E62366134E6732","metadata":[],"tags":[]}},"eventId":"1713628708.845630.b361fc92e0157fd7ee22ab33d627a3b7","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.649411Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6886667351242998518","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"41","size":3010700,"eTag":"3e722da8384f222b3753fa1948d2e9f2","versionId":"","sequencer":"24E62366054A6732","metadata":[],"tags":[]}},"eventId":"1713628708.845629.3e722da8384f222b3753fa1948d2e9f2","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.735017Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15339365163889935341","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"42","size":3013207,"eTag":"2f77136432f26bffbc861efbb992caac","versionId":"","sequencer":"24E62366F11E5F38","metadata":[],"tags":[]}},"eventId":"1713628708.945757.2f77136432f26bffbc861efbb992caac","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.734403Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17403901371603164130","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"43","size":3011046,"eTag":"4e88c1388d1b600beeda0fca24deb246","versionId":"","sequencer":"24E623668B3A5F38","metadata":[],"tags":[]}},"eventId":"1713628708.945765.4e88c1388d1b600beeda0fca24deb246","opaqueData":""}]}' 2024-04-20T16:10:03.245 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.778684Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.18340924620923969834","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"48","size":3010960,"eTag":"7bb352c63fbc72135a85e82033e05952","versionId":"","sequencer":"24E623660E7B6B38","metadata":[],"tags":[]}},"eventId":"1713628708.946567.7bb352c63fbc72135a85e82033e05952","opaqueData":""}]}' 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.792029Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14732378859543301633","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"47","size":3008151,"eTag":"567cebb91c3c18a1b84529102a129033","versionId":"","sequencer":"24E62366B0846B38","metadata":[],"tags":[]}},"eventId":"1713628708.946570.567cebb91c3c18a1b84529102a129033","opaqueData":""}]}' 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.828082Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6402583333206743309","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"50","size":3011009,"eTag":"9d8bfa9de75168ca2a22f70492b53014","versionId":"","sequencer":"24E623668AAE6B38","metadata":[],"tags":[]}},"eventId":"1713628708.946581.9d8bfa9de75168ca2a22f70492b53014","opaqueData":""}]}' 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.874236Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4174137785450450970","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"51","size":3011022,"eTag":"16285ba35ca00542d263390f8fa3bc1f","versionId":"","sequencer":"24E623667F1E6C38","metadata":[],"tags":[]}},"eventId":"1713628708.946609.16285ba35ca00542d263390f8fa3bc1f","opaqueData":""}]}' 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.844812Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15375269135918325354","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"49","size":3011063,"eTag":"79b2598dec8acc8db034c5d15c431866","versionId":"","sequencer":"24E623662FDD6B38","metadata":[],"tags":[]}},"eventId":"1713628708.946593.79b2598dec8acc8db034c5d15c431866","opaqueData":""}]}' 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.882616Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.14020226644781210991","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"54","size":3011708,"eTag":"08ac9d9ffccdd53f02ba84bb55ffd0c9","versionId":"","sequencer":"24E62366B1876C38","metadata":[],"tags":[]}},"eventId":"1713628708.946636.08ac9d9ffccdd53f02ba84bb55ffd0c9","opaqueData":""}]}' 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.246 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.882872Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.17258475958805296835","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"52","size":3010255,"eTag":"068beaa989d65be772ee9b4324b34d69","versionId":"","sequencer":"24E6236612876C38","metadata":[],"tags":[]}},"eventId":"1713628708.946636.068beaa989d65be772ee9b4324b34d69","opaqueData":""}]}' 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.903348Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12000835659748803139","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"55","size":3010361,"eTag":"db6e7e872ef51369424ad5799b57fcfc","versionId":"","sequencer":"24E623667DA06F38","metadata":[],"tags":[]}},"eventId":"1713628708.946839.db6e7e872ef51369424ad5799b57fcfc","opaqueData":""}]}' 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.888118Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8161191292885337777","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"53","size":3010204,"eTag":"b65b520b3971a039b68a78c8e3915f75","versionId":"","sequencer":"24E623667C7F6F38","metadata":[],"tags":[]}},"eventId":"1713628708.946831.b65b520b3971a039b68a78c8e3915f75","opaqueData":""}]}' 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/89 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/89 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/89 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue'} 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.247 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/86 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/86 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/86 2024-04-20T16:10:03.248 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue'} 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.249 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/88 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/88 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/88 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue'} 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/90 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/90 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/90 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue'} 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/91 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/91 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/91 2024-04-20T16:10:03.252 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue'} 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/92 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/92 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/92 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue'} 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/94 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/94 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/94 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue'} 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/93 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/93 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/93 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue'} 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.257 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/95 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/95 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/95 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue'} 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.258 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/97 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/97 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/97 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue'} 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.259 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/99 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/99 2024-04-20T16:10:03.260 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/99 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue'} 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.261 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/96 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/96 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/96 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue'} 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"068beaa989d65be772ee9b4324b34d69"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef826c71f93576c3-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b2598dec8acc8db034c5d15c431866"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d55fedd144705e6a-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"567cebb91c3c18a1b84529102a129033"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc73ec846b8d8e01-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"08ac9d9ffccdd53f02ba84bb55ffd0c9"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c291d9d9e0a69d6f-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"9d8bfa9de75168ca2a22f70492b53014"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058da8b9e849fd90d-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"4e88c1388d1b600beeda0fca24deb246"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f187141670571be2-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.263 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"16285ba35ca00542d263390f8fa3bc1f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039ed8433494f501a-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"2f77136432f26bffbc861efbb992caac"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4e05f5859a943ed-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7bb352c63fbc72135a85e82033e05952"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe880dc4b263a12a-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.753080Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15931644201860767922","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"46","size":3011653,"eTag":"5e064c8e7267fb5b9574fcdeb5234df1","versionId":"","sequencer":"24E6236635076E38","metadata":[],"tags":[]}},"eventId":"1713628708.946734.5e064c8e7267fb5b9574fcdeb5234df1","opaqueData":""}]}' 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.924255Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7809800318522890568","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"56","size":3010584,"eTag":"9f806c8ff887e1903eb2a961ef40c478","versionId":"","sequencer":"24E62366A6BD7038","metadata":[],"tags":[]}},"eventId":"1713628708.946912.9f806c8ff887e1903eb2a961ef40c478","opaqueData":""}]}' 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.933007Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7645736487297238199","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"57","size":3011645,"eTag":"ee96e54413d5be2ed6c1bafe8c37e2c1","versionId":"","sequencer":"24E62366B6487138","metadata":[],"tags":[]}},"eventId":"1713628708.946948.ee96e54413d5be2ed6c1bafe8c37e2c1","opaqueData":""}]}' 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"b65b520b3971a039b68a78c8e3915f75"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007142603db47106b1-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.264 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.961490Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13310796983848591842","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"59","size":3011792,"eTag":"1d6c2d3ba4019e23f75a89ccb640628b","versionId":"","sequencer":"24E62366BD5E093A","metadata":[],"tags":[]}},"eventId":"1713628708.973692.1d6c2d3ba4019e23f75a89ccb640628b","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.967720Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.899642995285920435","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"58","size":3009467,"eTag":"51c0c7d3ba9fae965e92377f25f6516d","versionId":"","sequencer":"24E62366C49A643A","metadata":[],"tags":[]}},"eventId":"1713628708.979671.51c0c7d3ba9fae965e92377f25f6516d","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.983777Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6231638925561761961","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"60","size":3009879,"eTag":"a8002e99d53f892b085de725a05a8a9d","versionId":"","sequencer":"24E62366383C333B","metadata":[],"tags":[]}},"eventId":"1713628708.993213.a8002e99d53f892b085de725a05a8a9d","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.996545Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1109850757163131334","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"62","size":3008324,"eTag":"fad3a3c952c262dead1ac82897fbb7ed","versionId":"","sequencer":"25E62366198C8400","metadata":[],"tags":[]}},"eventId":"1713628709.008686.fad3a3c952c262dead1ac82897fbb7ed","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.001716Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10497749439508854960","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"61","size":3009352,"eTag":"8575fa3c93e6e5e8116b8e58f85b65b8","versionId":"","sequencer":"25E62366B123DC00","metadata":[],"tags":[]}},"eventId":"1713628709.014427.8575fa3c93e6e5e8116b8e58f85b65b8","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.004079Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16439601104106800976","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"63","size":3013115,"eTag":"6f97dd4ad4a0545211afdc13288df53b","versionId":"","sequencer":"25E62366D3925D01","metadata":[],"tags":[]}},"eventId":"1713628709.022909.6f97dd4ad4a0545211afdc13288df53b","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.008912Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4075445044848366673","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"64","size":3007840,"eTag":"019a7be19c7cf4b2de53ea8b43c55a40","versionId":"","sequencer":"25E623664A836201","metadata":[],"tags":[]}},"eventId":"1713628709.023233.019a7be19c7cf4b2de53ea8b43c55a40","opaqueData":""}]}' 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.265 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.022058Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13556555462519659260","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"65","size":3009881,"eTag":"94df7113e9919325f74bf6d2439332fc","versionId":"","sequencer":"25E623662170EA01","metadata":[],"tags":[]}},"eventId":"1713628709.032141.94df7113e9919325f74bf6d2439332fc","opaqueData":""}]}' 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.024302Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12876205884765558987","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"66","size":3010129,"eTag":"f4bf481a2c2338eefa1945558ffb0276","versionId":"","sequencer":"25E6236665473402","metadata":[],"tags":[]}},"eventId":"1713628709.036980.f4bf481a2c2338eefa1945558ffb0276","opaqueData":""}]}' 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/98 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/98 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/98 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue'} 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.266 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"db6e7e872ef51369424ad5799b57fcfc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a68b88730b2b0243-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.032038Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6818035508816379486","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"68","size":3010165,"eTag":"d7a10a56fbc47feb396529a1fa688676","versionId":"","sequencer":"25E623661A954E03","metadata":[],"tags":[]}},"eventId":"1713628709.055481.d7a10a56fbc47feb396529a1fa688676","opaqueData":""}]}' 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.267 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.041265Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.3465071374219490384","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"67","size":3010401,"eTag":"e96da24ee0ef7da1ef453dc3d6fc9316","versionId":"","sequencer":"25E62366FCBB6403","metadata":[],"tags":[]}},"eventId":"1713628709.056933.e96da24ee0ef7da1ef453dc3d6fc9316","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.064560Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13100248896837074962","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"69","size":3008237,"eTag":"5b2cb66578f1c3e98159a2ea4d842c43","versionId":"","sequencer":"25E623669D3A9704","metadata":[],"tags":[]}},"eventId":"1713628709.077019.5b2cb66578f1c3e98159a2ea4d842c43","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.080878Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.18336613288276755873","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"71","size":3008799,"eTag":"a10b357f5acac3fac7a15c552d99032c","versionId":"","sequencer":"25E623661688E205","metadata":[],"tags":[]}},"eventId":"1713628709.098732.a10b357f5acac3fac7a15c552d99032c","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.088161Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15929575312809232798","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"72","size":3012871,"eTag":"b859d03084fcfd4f532f1e960ac26fcb","versionId":"","sequencer":"25E6236620C74606","metadata":[],"tags":[]}},"eventId":"1713628709.105301.b859d03084fcfd4f532f1e960ac26fcb","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.083537Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8417420348105737735","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"73","size":3011501,"eTag":"4d20dc10900f921ad11ae05208c8cbf3","versionId":"","sequencer":"25E623664F368F06","metadata":[],"tags":[]}},"eventId":"1713628709.110048.4d20dc10900f921ad11ae05208c8cbf3","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.088161Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8074119970030431844","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"70","size":3011693,"eTag":"0f57dc487333430f7108d6e44d89f2d1","versionId":"","sequencer":"25E62366A1D1DB06","metadata":[],"tags":[]}},"eventId":"1713628709.115069.0f57dc487333430f7108d6e44d89f2d1","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.092897Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6674351261468416344","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"74","size":3011849,"eTag":"04ecd4690ee5e8f83a588768713a91ba","versionId":"","sequencer":"25E62366BCAAE106","metadata":[],"tags":[]}},"eventId":"1713628709.115452.04ecd4690ee5e8f83a588768713a91ba","opaqueData":""}]}' 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.268 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.094715Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.122826133796831472","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"75","size":3008117,"eTag":"6d1d8682beb009e593a0840380993800","versionId":"","sequencer":"25E623662449DE06","metadata":[],"tags":[]}},"eventId":"1713628709.115231.6d1d8682beb009e593a0840380993800","opaqueData":""}]}' 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.103014Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15930250310856652464","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"78","size":3011218,"eTag":"1754738b32468649af3de5315edf5908","versionId":"","sequencer":"25E623660A2CBC07","metadata":[],"tags":[]}},"eventId":"1713628709.129772.1754738b32468649af3de5315edf5908","opaqueData":""}]}' 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.111918Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7466113218565974825","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"81","size":3008315,"eTag":"0afe30bff8b47c853b0207a1c2811e7b","versionId":"","sequencer":"25E62366E41BBA07","metadata":[],"tags":[]}},"eventId":"1713628709.129637.0afe30bff8b47c853b0207a1c2811e7b","opaqueData":""}]}' 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.127897Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15242081170700100356","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"77","size":3011971,"eTag":"ed15f2a61fc86bb8dbeb38b396a68115","versionId":"","sequencer":"25E62366B2794F08","metadata":[],"tags":[]}},"eventId":"1713628709.139426.ed15f2a61fc86bb8dbeb38b396a68115","opaqueData":""}]}' 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"5e064c8e7267fb5b9574fcdeb5234df1"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd1891fdfbd2a0b2-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"9f806c8ff887e1903eb2a961ef40c478"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c61fc02f84a1948-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"ee96e54413d5be2ed6c1bafe8c37e2c1"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a1b1cd38b9844b7-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.123254Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.354795825593411728","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"79","size":3011496,"eTag":"6f8b16f076b343e51d81915416c9d417","versionId":"","sequencer":"25E623661215C908","metadata":[],"tags":[]}},"eventId":"1713628709.147395.6f8b16f076b343e51d81915416c9d417","opaqueData":""}]}' 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.269 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.961490Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13310796983848591842","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"59","size":3011792,"eTag":"1d6c2d3ba4019e23f75a89ccb640628b","versionId":"","sequencer":"25E62366B6963309","metadata":[],"tags":[]}},"eventId":"1713628709.154375.1d6c2d3ba4019e23f75a89ccb640628b","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.967720Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.899642995285920435","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"58","size":3009467,"eTag":"51c0c7d3ba9fae965e92377f25f6516d","versionId":"","sequencer":"25E623662C923409","metadata":[],"tags":[]}},"eventId":"1713628709.154440.51c0c7d3ba9fae965e92377f25f6516d","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.983777Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6231638925561761961","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"60","size":3009879,"eTag":"a8002e99d53f892b085de725a05a8a9d","versionId":"","sequencer":"25E6236619A63409","metadata":[],"tags":[]}},"eventId":"1713628709.154445.a8002e99d53f892b085de725a05a8a9d","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.001716Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10497749439508854960","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"61","size":3009352,"eTag":"8575fa3c93e6e5e8116b8e58f85b65b8","versionId":"","sequencer":"25E623660D2B3509","metadata":[],"tags":[]}},"eventId":"1713628709.154479.8575fa3c93e6e5e8116b8e58f85b65b8","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.004079Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.16439601104106800976","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"63","size":3013115,"eTag":"6f97dd4ad4a0545211afdc13288df53b","versionId":"","sequencer":"25E623664C5B3509","metadata":[],"tags":[]}},"eventId":"1713628709.154491.6f97dd4ad4a0545211afdc13288df53b","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.008912Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.4075445044848366673","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"64","size":3007840,"eTag":"019a7be19c7cf4b2de53ea8b43c55a40","versionId":"","sequencer":"25E6236644A33509","metadata":[],"tags":[]}},"eventId":"1713628709.154510.019a7be19c7cf4b2de53ea8b43c55a40","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.022058Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13556555462519659260","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"65","size":3009881,"eTag":"94df7113e9919325f74bf6d2439332fc","versionId":"","sequencer":"25E62366B8E13509","metadata":[],"tags":[]}},"eventId":"1713628709.154526.94df7113e9919325f74bf6d2439332fc","opaqueData":""}]}' 2024-04-20T16:10:03.270 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.024302Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12876205884765558987","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"66","size":3010129,"eTag":"f4bf481a2c2338eefa1945558ffb0276","versionId":"","sequencer":"25E62366AC013609","metadata":[],"tags":[]}},"eventId":"1713628709.154534.f4bf481a2c2338eefa1945558ffb0276","opaqueData":""}]}' 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.032038Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6818035508816379486","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"68","size":3010165,"eTag":"d7a10a56fbc47feb396529a1fa688676","versionId":"","sequencer":"25E62366D7383609","metadata":[],"tags":[]}},"eventId":"1713628709.154548.d7a10a56fbc47feb396529a1fa688676","opaqueData":""}]}' 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.041265Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.3465071374219490384","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"67","size":3010401,"eTag":"e96da24ee0ef7da1ef453dc3d6fc9316","versionId":"","sequencer":"25E62366BA813609","metadata":[],"tags":[]}},"eventId":"1713628709.154567.e96da24ee0ef7da1ef453dc3d6fc9316","opaqueData":""}]}' 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.064560Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13100248896837074962","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"69","size":3008237,"eTag":"5b2cb66578f1c3e98159a2ea4d842c43","versionId":"","sequencer":"25E6236630C83609","metadata":[],"tags":[]}},"eventId":"1713628709.154585.5b2cb66578f1c3e98159a2ea4d842c43","opaqueData":""}]}' 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.080878Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.18336613288276755873","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"71","size":3008799,"eTag":"a10b357f5acac3fac7a15c552d99032c","versionId":"","sequencer":"25E62366E7E93609","metadata":[],"tags":[]}},"eventId":"1713628709.154593.a10b357f5acac3fac7a15c552d99032c","opaqueData":""}]}' 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.083537Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8417420348105737735","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"73","size":3011501,"eTag":"4d20dc10900f921ad11ae05208c8cbf3","versionId":"","sequencer":"25E623660C433709","metadata":[],"tags":[]}},"eventId":"1713628709.154616.4d20dc10900f921ad11ae05208c8cbf3","opaqueData":""}]}' 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.271 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.092897Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6674351261468416344","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"74","size":3011849,"eTag":"04ecd4690ee5e8f83a588768713a91ba","versionId":"","sequencer":"25E6236601883709","metadata":[],"tags":[]}},"eventId":"1713628709.154634.04ecd4690ee5e8f83a588768713a91ba","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.088161Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8074119970030431844","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"70","size":3011693,"eTag":"0f57dc487333430f7108d6e44d89f2d1","versionId":"","sequencer":"25E62366B7873709","metadata":[],"tags":[]}},"eventId":"1713628709.154634.0f57dc487333430f7108d6e44d89f2d1","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.103014Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15930250310856652464","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"78","size":3011218,"eTag":"1754738b32468649af3de5315edf5908","versionId":"","sequencer":"25E6236681FE3709","metadata":[],"tags":[]}},"eventId":"1713628709.154664.1754738b32468649af3de5315edf5908","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.111918Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.7466113218565974825","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"81","size":3008315,"eTag":"0afe30bff8b47c853b0207a1c2811e7b","versionId":"","sequencer":"25E62366B0583809","metadata":[],"tags":[]}},"eventId":"1713628709.154687.0afe30bff8b47c853b0207a1c2811e7b","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.127897Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15242081170700100356","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"77","size":3011971,"eTag":"ed15f2a61fc86bb8dbeb38b396a68115","versionId":"","sequencer":"25E6236649823809","metadata":[],"tags":[]}},"eventId":"1713628709.154698.ed15f2a61fc86bb8dbeb38b396a68115","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.088161Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15929575312809232798","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"72","size":3012871,"eTag":"b859d03084fcfd4f532f1e960ac26fcb","versionId":"","sequencer":"25E623662E163709","metadata":[],"tags":[]}},"eventId":"1713628709.154605.b859d03084fcfd4f532f1e960ac26fcb","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.094715Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.122826133796831472","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"75","size":3008117,"eTag":"6d1d8682beb009e593a0840380993800","versionId":"","sequencer":"25E6236638D73709","metadata":[],"tags":[]}},"eventId":"1713628709.154654.6d1d8682beb009e593a0840380993800","opaqueData":""}]}' 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.272 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:28.996545Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1109850757163131334","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"62","size":3008324,"eTag":"fad3a3c952c262dead1ac82897fbb7ed","versionId":"","sequencer":"25E6236605BD3409","metadata":[],"tags":[]}},"eventId":"1713628709.154451.fad3a3c952c262dead1ac82897fbb7ed","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.121621Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.11591403144806841447","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"80","size":3013569,"eTag":"25cae1324a904c7e6ee581eb5943f172","versionId":"","sequencer":"25E623665371CF08","metadata":[],"tags":[]}},"eventId":"1713628709.147812.25cae1324a904c7e6ee581eb5943f172","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.123532Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12697869617350863449","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"76","size":3010726,"eTag":"051d4ae802d25782c3ef038533dfebcc","versionId":"","sequencer":"25E62366D716E908","metadata":[],"tags":[]}},"eventId":"1713628709.149493.051d4ae802d25782c3ef038533dfebcc","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.140523Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6821006023785673431","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"82","size":3008911,"eTag":"45e807a018f296a2fe94353d27a5e5e7","versionId":"","sequencer":"25E62366F049F908","metadata":[],"tags":[]}},"eventId":"1713628709.150555.45e807a018f296a2fe94353d27a5e5e7","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.148238Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9179506120996517248","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"85","size":3011397,"eTag":"00af8114831da10747c9e1230815f096","versionId":"","sequencer":"25E62366E3BE160B","metadata":[],"tags":[]}},"eventId":"1713628709.186040.00af8114831da10747c9e1230815f096","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.153426Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.11250647941318573350","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"83","size":3009084,"eTag":"bd0a3d3ee78188aa3ef0e1df20e459c6","versionId":"","sequencer":"25E623669F0D7F0B","metadata":[],"tags":[]}},"eventId":"1713628709.192875.bd0a3d3ee78188aa3ef0e1df20e459c6","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.177263Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1580857053642427182","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"93","size":3012826,"eTag":"444c569a60d54727c3f18efae5534066","versionId":"","sequencer":"25E623669471910B","metadata":[],"tags":[]}},"eventId":"1713628709.194081.444c569a60d54727c3f18efae5534066","opaqueData":""}]}' 2024-04-20T16:10:03.273 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.163990Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.2660928233041935798","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"89","size":3010889,"eTag":"56bbf4e8fbdfa807b7e7311196fd7ff6","versionId":"","sequencer":"25E62366512D7E0B","metadata":[],"tags":[]}},"eventId":"1713628709.192818.56bbf4e8fbdfa807b7e7311196fd7ff6","opaqueData":""}]}' 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.159303Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9407815226470704287","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"84","size":3009841,"eTag":"6787d7556242ff5dd62a0eabc04aa7c1","versionId":"","sequencer":"25E62366B195830B","metadata":[],"tags":[]}},"eventId":"1713628709.193172.6787d7556242ff5dd62a0eabc04aa7c1","opaqueData":""}]}' 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.163981Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10373502053404636357","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"88","size":3009573,"eTag":"05f9b4ff64bfba06ae9df272132ef06d","versionId":"","sequencer":"25E623666B5D930B","metadata":[],"tags":[]}},"eventId":"1713628709.194207.05f9b4ff64bfba06ae9df272132ef06d","opaqueData":""}]}' 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.177851Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9825244058134919691","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"94","size":3011688,"eTag":"0aacd8a6312ac51d82b84b70add458aa","versionId":"","sequencer":"25E62366672A180C","metadata":[],"tags":[]}},"eventId":"1713628709.202910.0aacd8a6312ac51d82b84b70add458aa","opaqueData":""}]}' 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.177439Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13855577106429474243","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"98","size":3009922,"eTag":"53827da56141f2c4ab1957e8264483d7","versionId":"","sequencer":"25E623668DD31F0C","metadata":[],"tags":[]}},"eventId":"1713628709.203412.53827da56141f2c4ab1957e8264483d7","opaqueData":""}]}' 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.183502Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.13678364157846785268","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"97","size":3008275,"eTag":"79ebe7b711dfe1723475409625af5b58","versionId":"","sequencer":"25E6236628C0560D","metadata":[],"tags":[]}},"eventId":"1713628709.223789.79ebe7b711dfe1723475409625af5b58","opaqueData":""}]}' 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.274 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.188326Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15465991643338641271","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"99","size":3008985,"eTag":"677f5b3cafb3c801bf96f6cc466866a7","versionId":"","sequencer":"25E62366E3405E0D","metadata":[],"tags":[]}},"eventId":"1713628709.224280.677f5b3cafb3c801bf96f6cc466866a7","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.188531Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10096929148714101898","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"95","size":3011691,"eTag":"006d0fc2abc804f8d6825babf5f3611b","versionId":"","sequencer":"25E62366C7B85D0D","metadata":[],"tags":[]}},"eventId":"1713628709.224245.006d0fc2abc804f8d6825babf5f3611b","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.177307Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.2853672845506860322","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"90","size":3011398,"eTag":"9ed43938d6d328844799d3cc295e54bb","versionId":"","sequencer":"25E62366B99F630D","metadata":[],"tags":[]}},"eventId":"1713628709.224632.9ed43938d6d328844799d3cc295e54bb","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.171064Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.10841161051665124892","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"86","size":3010244,"eTag":"39125ed67862b2db0f379e9309f6e89e","versionId":"","sequencer":"25E623664ABE670D","metadata":[],"tags":[]}},"eventId":"1713628709.224902.39125ed67862b2db0f379e9309f6e89e","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.178004Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.8012778382684611019","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"92","size":3010920,"eTag":"39b57e822e55abef62e0a1eaf2761e98","versionId":"","sequencer":"25E62366158D7A0D","metadata":[],"tags":[]}},"eventId":"1713628709.226135.39b57e822e55abef62e0a1eaf2761e98","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.172798Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.3557145103957421102","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"91","size":3008594,"eTag":"638f210dd9127b69ec8fcba42a256de5","versionId":"","sequencer":"25E623667E527B0D","metadata":[],"tags":[]}},"eventId":"1713628709.226185.638f210dd9127b69ec8fcba42a256de5","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.168223Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.15414220664494760687","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"87","size":3008741,"eTag":"e0532dd4183ac7191f3a73c10d1be5be","versionId":"","sequencer":"25E62366F7E2A20D","metadata":[],"tags":[]}},"eventId":"1713628709.228778.e0532dd4183ac7191f3a73c10d1be5be","opaqueData":""}]}' 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_1: 2024-04-20T16:10:03.275 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.188422Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9950982043631153701","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_1","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"96","size":3010669,"eTag":"c397f4c401734701fa2687fd49f6672a","versionId":"","sequencer":"25E623661A2BA50D","metadata":[],"tags":[]}},"eventId":"1713628709.228928.c397f4c401734701fa2687fd49f6672a","opaqueData":""}]}' 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"94df7113e9919325f74bf6d2439332fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc228f5e115d9afc-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"1d6c2d3ba4019e23f75a89ccb640628b"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8b973554ba2e1e2-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"51c0c7d3ba9fae965e92377f25f6516d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c7c2c89a6af8ab3-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"1754738b32468649af3de5315edf5908"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd139e4171aab6b0-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"019a7be19c7cf4b2de53ea8b43c55a40"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000388ee3acf7d01451-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"d7a10a56fbc47feb396529a1fa688676"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e9e872bf8fffe5e-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"ed15f2a61fc86bb8dbeb38b396a68115"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d386c011993fe304-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8002e99d53f892b085de725a05a8a9d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000567b3a960e9b10a9-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"f4bf481a2c2338eefa1945558ffb0276"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2b17907f83ed8cb-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"5b2cb66578f1c3e98159a2ea4d842c43"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5cd6ef6ad2d4012-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"0afe30bff8b47c853b0207a1c2811e7b"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000679cf66a45792b29-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"4d20dc10900f921ad11ae05208c8cbf3"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074d0af358dfcc607-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"0f57dc487333430f7108d6e44d89f2d1"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000700d09558f8d2e64-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.276 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a10b357f5acac3fac7a15c552d99032c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe78bca2577185a1-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"8575fa3c93e6e5e8116b8e58f85b65b8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091af7f7eeafb38b0-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.123254Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.354795825593411728","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"79","size":3011496,"eTag":"6f8b16f076b343e51d81915416c9d417","versionId":"","sequencer":"25E62366C944380F","metadata":[],"tags":[]}},"eventId":"1713628709.255345.6f8b16f076b343e51d81915416c9d417","opaqueData":""}]}' 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.123532Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.12697869617350863449","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"76","size":3010726,"eTag":"051d4ae802d25782c3ef038533dfebcc","versionId":"","sequencer":"25E62366D76B3115","metadata":[],"tags":[]}},"eventId":"1713628709.355560.051d4ae802d25782c3ef038533dfebcc","opaqueData":""}]}' 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.121621Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.11591403144806841447","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"80","size":3013569,"eTag":"25cae1324a904c7e6ee581eb5943f172","versionId":"","sequencer":"25E62366D8483115","metadata":[],"tags":[]}},"eventId":"1713628709.355551.25cae1324a904c7e6ee581eb5943f172","opaqueData":""}]}' 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.140523Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.6821006023785673431","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"82","size":3008911,"eTag":"45e807a018f296a2fe94353d27a5e5e7","versionId":"","sequencer":"25E62366E4D33615","metadata":[],"tags":[]}},"eventId":"1713628709.355914.45e807a018f296a2fe94353d27a5e5e7","opaqueData":""}]}' 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.153426Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.11250647941318573350","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"83","size":3009084,"eTag":"bd0a3d3ee78188aa3ef0e1df20e459c6","versionId":"","sequencer":"25E62366915D3715","metadata":[],"tags":[]}},"eventId":"1713628709.355949.bd0a3d3ee78188aa3ef0e1df20e459c6","opaqueData":""}]}' 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.148238Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.9179506120996517248","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"85","size":3011397,"eTag":"00af8114831da10747c9e1230815f096","versionId":"","sequencer":"25E62366E7303715","metadata":[],"tags":[]}},"eventId":"1713628709.355938.00af8114831da10747c9e1230815f096","opaqueData":""}]}' 2024-04-20T16:10:03.277 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.177263Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.1580857053642427182","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"93","size":3012826,"eTag":"444c569a60d54727c3f18efae5534066","versionId":"","sequencer":"25E62366376A3715","metadata":[],"tags":[]}},"eventId":"1713628709.355953.444c569a60d54727c3f18efae5534066","opaqueData":""}]}' 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:bucket_notification.test_bn: INFO: AMQP received event for topic lbboah-2_topic_2: 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr: b'{"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2024-04-20T15:58:29.163990Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"foo.client.0"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4521.2660928233041935798","x-amz-id-2":"11a9-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"lbboah-2_notif_2","bucket":{"name":"lbboah-2","ownerIdentity":{"principalId":"foo.client.0"},"arn":"arn:aws:s3:::lbboah-2","id":"f76a52bc-49fc-4b9b-bb36-9008f7b8b677.4451.2"},"object":{"key":"89","size":3010889,"eTag":"56bbf4e8fbdfa807b7e7311196fd7ff6","versionId":"","sequencer":"25E62366B2423915","metadata":[],"tags":[]}},"eventId":"1713628709.356074.56bbf4e8fbdfa807b7e7311196fd7ff6","opaqueData":""}]}' 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"b859d03084fcfd4f532f1e960ac26fcb"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd11385936d3899e-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"fad3a3c952c262dead1ac82897fbb7ed"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f66fb62266821c6-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"04ecd4690ee5e8f83a588768713a91ba"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ca00f1c2e015958-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6f8b16f076b343e51d81915416c9d417"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004ec7cec70496890-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"e96da24ee0ef7da1ef453dc3d6fc9316"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003016680e5dba3450-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6d1d8682beb009e593a0840380993800"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001b45db7ca1840f0-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.278 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"6f97dd4ad4a0545211afdc13288df53b"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e425321a7b67c750-006623e624-11a9-default'), ('Date', 'Sat, 20 Apr 2024 15:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.279 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.280 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered RemoteDisconnected exception, reconnecting 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.281 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.282 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.283 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.284 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.285 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.286 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.287 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.288 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.289 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.290 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.291 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.292 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.293 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.295 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.296 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.297 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.298 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.299 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.300 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.301 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.302 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.303 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.304 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.305 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.306 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.307 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.310 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.318 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.326 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.328 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.330 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.331 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.333 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.334 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.335 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.336 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.337 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.338 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.339 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.341 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.342 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.343 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.344 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.345 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.346 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.347 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.348 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.349 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.350 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.351 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.352 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.353 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.354 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.355 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.356 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.357 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.358 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:4FMt1Bg6xxkfOnPBDRvlvg== 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/87 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k= 2024-04-20T16:10:03.359 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': '4FMt1Bg6xxkfOnPBDRvlvg==', 'Content-Length': '3008741', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:PfLGEwrObXIckyykKsQzAZ/ak4k='} 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:Y48hDdkSe2nsj8ukKiVt5Q== 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/91 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo= 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Y48hDdkSe2nsj8ukKiVt5Q==', 'Content-Length': '3008594', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:5Vp75FS8m7gYy9/+/pVnuGF8UPo='} 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.360 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.361 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.362 INFO:teuthology.orchestra.run.smithi071.stderr:Vrv06PvfqAe35zERlv1/9g== 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/89 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg= 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Vrv06PvfqAe35zERlv1/9g==', 'Content-Length': '3010889', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:BVKXSSL1Z7dOOAxzF/lFR9vg4tg='} 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.363 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.364 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:w5f0xAFzRwH6Jof9SfZnKg== 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/96 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI= 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'w5f0xAFzRwH6Jof9SfZnKg==', 'Content-Length': '3010669', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:XyYBughA4h7tZpAb8A5+dgI5nvI='} 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.365 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.366 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:AK+BFIMdoQdHyeEjCBXwlg== 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/85 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w= 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AK+BFIMdoQdHyeEjCBXwlg==', 'Content-Length': '3011397', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:cGR3jHlmcYcYY9T0MX9xHzmQb5w='} 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:eevntxHf4XI0dUCWJa9bWA== 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/97 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0= 2024-04-20T16:10:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'eevntxHf4XI0dUCWJa9bWA==', 'Content-Length': '3008275', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:n+lrMCpP6VffBQt/K793zX/nhI0='} 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:ORJe1nhistsPN56TCfbong== 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/86 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4= 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ORJe1nhistsPN56TCfbong==', 'Content-Length': '3010244', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Rrw0qe2Ywl4crX1AtyijK4BNIY4='} 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.369 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.370 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=914.563426357 and callback=>; now=884.563426357; delay=30.0 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.371 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=914.575144531 and callback=>; now=884.575144531; delay=30.0 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:vQo9PueBiKo+8OHfIORZxg== 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/83 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.372 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk= 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'vQo9PueBiKo+8OHfIORZxg==', 'Content-Length': '3009084', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:I1lL+HYrTYm/SBcLPIdXjMXDhmk='} 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:Bfm0/2S/ugaunfJyEy7wbQ== 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/88 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg= 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Bfm0/2S/ugaunfJyEy7wbQ==', 'Content-Length': '3009573', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:HE/3m2bWhZSuWbSUAyWIlYe03qg='} 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.373 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:ObV+gi5Vq+9i4KHq8nYemA== 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/92 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak= 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ObV+gi5Vq+9i4KHq8nYemA==', 'Content-Length': '3010920', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:f4Iin2XFHrLkZVJdr+aGDLgoYak='} 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.374 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:CqzYpjEqxR2CuEtwrdRYqg== 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/94 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI= 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'CqzYpjEqxR2CuEtwrdRYqg==', 'Content-Length': '3011688', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vKl2wrumd9AyeRgGs6zRoMxK1dI='} 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:RegHoBjylqL+lDU9J6Xl5w== 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.375 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/82 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU= 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RegHoBjylqL+lDU9J6Xl5w==', 'Content-Length': '3008911', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:oMjbiZcDjnsTedoCyrwBmtjEfIU='} 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:BR1K6ALSV4LD7wOFM9/rzA== 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/76 2024-04-20T16:10:03.376 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs= 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'BR1K6ALSV4LD7wOFM9/rzA==', 'Content-Length': '3010726', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:N1X/HWXQKsoIcyju+WaKgjLUQIs='} 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:JcrhMkqQTH5u5YHrWUPxcg== 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/80 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ= 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'JcrhMkqQTH5u5YHrWUPxcg==', 'Content-Length': '3013569', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:thWM2bgXYpvz8bxwIN9BzZQIOpQ='} 2024-04-20T16:10:03.377 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:ntQ5ONbTKIRHmdPMKV5Uuw== 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/90 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY= 2024-04-20T16:10:03.378 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ntQ5ONbTKIRHmdPMKV5Uuw==', 'Content-Length': '3011398', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:3URqyphX3xm612Yyo/vEINls3QY='} 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:RExWmmDVRyfD8Y765VNAZg== 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/93 2024-04-20T16:10:03.381 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE= 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'RExWmmDVRyfD8Y765VNAZg==', 'Content-Length': '3012826', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Uqa1yvETWOBuysczKNMNvhpYMIE='} 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:AG0PwqvIBPjWglur9fNhGw== 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/95 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M= 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'AG0PwqvIBPjWglur9fNhGw==', 'Content-Length': '3011691', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:/fSMk48B++nLmuT3hrULBz0rR9M='} 2024-04-20T16:10:03.382 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:Z39bPK+zyAG/lvbMRmhmpw== 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/99 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8= 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z39bPK+zyAG/lvbMRmhmpw==', 'Content-Length': '3008985', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:KW83TpFj13Dnjn6InQ269cuoqw8='} 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.383 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:U4J9pWFB8sSrGVfoJkSD1w== 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:29 GMT 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/98 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4= 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'U4J9pWFB8sSrGVfoJkSD1w==', 'Content-Length': '3009922', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:29 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:G8OrtNzpUV59Pt8VjBYHUUqNNJ4='} 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:Z4fXVWJC/13WKg6rwEqnwQ== 2024-04-20T16:10:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:application/octet-stream 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 15:58:28 GMT 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/84 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ= 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Z4fXVWJC/13WKg6rwEqnwQ==', 'Content-Length': '3009841', 'Expect': '100-Continue', 'Date': 'Sat, 20 Apr 2024 15:58:28 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:DO77t3A/Iqve1Nlbowhv3UeGXbQ='} 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=944.564455411 and callback=>; now=914.564455411; delay=30.0 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=944.576304539 and callback=>; now=914.576304539; delay=30.0 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.386 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 2 heartbeat frames, sent 2, idle intervals 0 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=984.567031022 and callback=>; now=919.567031022; delay=65 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 2 heartbeat frames, sent 2, idle intervals 0 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=984.57810137 and callback=>; now=919.57810137; delay=65 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=974.56537773 and callback=>; now=944.56537773; delay=30.0 2024-04-20T16:10:03.387 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=974.577188916 and callback=>; now=944.577188916; delay=30.0 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-2/ 2024-04-20T16:10:03.388 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-2/ 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: GET 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-2/ 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.390 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.391 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.392 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.393 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.394 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1004.566454209 and callback=>; now=974.566454209; delay=30.0 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1004.578153435 and callback=>; now=974.578153435; delay=30.0 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.395 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 4 heartbeat frames, sent 4, idle intervals 0 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1049.568886569 and callback=>; now=984.568886569; delay=65 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 4 heartbeat frames, sent 4, idle intervals 0 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1049.583604076 and callback=>; now=984.583604076; delay=65 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:GET 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:00:09 GMT 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-2/ 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE= 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:00:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:4GtU9SVzo8T4SW5TPOkqMNDGjtE=', 'Content-Length': '0'} 2024-04-20T16:10:03.396 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1034.5673787169999 and callback=>; now=1004.567378717; delay=30.0 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.397 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1034.5792999770001 and callback=>; now=1004.579299977; delay=30.0 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.398 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1064.568267452 and callback=>; now=1034.568267452; delay=30.0 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1064.580298778 and callback=>; now=1034.580298778; delay=30.0 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: bucket_notification.test_bn.test_ps_s3_creation_triggers_on_master 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 1859, in test_ps_s3_creation_triggers_on_master 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: ps_s3_creation_triggers_on_master() 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 1783, in ps_s3_creation_triggers_on_master 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.400 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.401 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.402 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-3/ 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-3/ 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-3/ 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.403 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.404 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.405 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 6 heartbeat frames, sent 6, idle intervals 0 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1114.575086965 and callback=>; now=1049.575086965; delay=65 2024-04-20T16:10:03.407 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 6 heartbeat frames, sent 6, idle intervals 0 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1114.589197632 and callback=>; now=1049.589197632; delay=65 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.408 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1094.569313053 and callback=>; now=1064.569313053; delay=30.0 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.409 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1094.581190608 and callback=>; now=1064.581190608; delay=30.0 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:01:32 GMT 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-3/ 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.410 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s= 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:01:32 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:hTYFdMcK8CzeAb1Kf/pKoksJm6s=', 'Content-Length': '0'} 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1124.570426262 and callback=>; now=1094.570426262; delay=30.0 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.411 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1124.582230423 and callback=>; now=1094.582230423; delay=30.0 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 8 heartbeat frames, sent 8, idle intervals 0 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1179.58048865 and callback=>; now=1114.58048865; delay=65 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 8 heartbeat frames, sent 8, idle intervals 0 2024-04-20T16:10:03.412 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1179.59466311 and callback=>; now=1114.59466311; delay=65 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test that uploads object using HTTP POST 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2026, in test_http_post_object_upload 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn1.create_bucket(ACL='public-read-write', Bucket=bucket_name) 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/client.py", line 508, in _api_call 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: return self._make_api_call(operation_name, kwargs) 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/client.py", line 895, in _make_api_call 2024-04-20T16:10:03.413 INFO:teuthology.orchestra.run.smithi071.stderr: operation_model, request_dict, request_context 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/client.py", line 917, in _make_request 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: return self._endpoint.make_request(operation_model, request_dict) 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 116, in make_request 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: return self._send_request(request_dict, operation_model) 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 204, in _send_request 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: exception, 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 358, in _needs_retry 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: request_dict=request_dict, 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/hooks.py", line 412, in emit 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: return self._emitter.emit(aliased_event_name, **kwargs) 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/hooks.py", line 256, in emit 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: return self._emit(event_name, kwargs) 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/hooks.py", line 239, in _emit 2024-04-20T16:10:03.414 INFO:teuthology.orchestra.run.smithi071.stderr: response = handler(**kwargs) 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 207, in __call__ 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: if self._checker(**checker_kwargs): 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 285, in __call__ 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 320, in _should_retry 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: return self._checker(attempt_number, response, caught_exception) 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 364, in __call__ 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 248, in __call__ 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, caught_exception 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: raise caught_exception 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.415 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7fd0cea10620> 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.PutObject: calling handler > 2024-04-20T16:10:03.416 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.PutObject: calling handler > 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/lbboah-4', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read-write', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-552.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi071.front.sepia.ceph.com:80/lbboah-4', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'lbboah-4'}}} 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.417 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-4 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-acl:public-read-write 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T160256Z 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.418 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T160256Z 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/us-east-1/s3/aws4_request 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:054ff131c100744e4dc344891bbbfa32e10672fe067e7cfe48e93d1b36ebc6c8 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:099c9e8dc34b3b6d10e8b746a62d919b87906338cdd2e2119350bfcf86d632d0 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Exception received when sending HTTP request. 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.419 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.420 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.421 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.422 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.422 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.422 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.422 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.422 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.424 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: retry needed, retryable exception caught: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.425 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.426 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.427 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.428 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 308, in _should_retry 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 364, in __call__ 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 248, in __call__ 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, caught_exception 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: raise caught_exception 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.429 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: Retry needed, action of: 0.6492401674927916 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Response received to retry, sleeping for 0.6492401674927916 seconds 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1154.571281437 and callback=>; now=1124.571281437; delay=30.0 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.430 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1154.583259431 and callback=>; now=1124.583259431; delay=30.0 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.431 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-4 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-acl:public-read-write 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T160257Z 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.432 INFO:teuthology.orchestra.run.smithi071.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T160257Z 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/us-east-1/s3/aws4_request 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:23fc78136ab7204ea79d14c425ef127d285b763cdadf57a8f0ffcae38b4c959a 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:e3f46dd5772a80bb97357e084a5e514df3a7294a9264d3dfb53e1803378036e3 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (2): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Exception received when sending HTTP request. 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.433 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.434 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.435 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.436 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.437 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: retry needed, retryable exception caught: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.438 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.439 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.440 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 308, in _should_retry 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 364, in __call__ 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 248, in __call__ 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, caught_exception 2024-04-20T16:10:03.441 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: raise caught_exception 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: Retry needed, action of: 1.3802454884980933 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Response received to retry, sleeping for 1.3802454884980933 seconds 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.442 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-4 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-acl:public-read-write 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T160258Z 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.443 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T160258Z 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/us-east-1/s3/aws4_request 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:e95fd94d77d48244822cf1918e47e3cc263e1a4d53ca5c7a32d1451820be23be 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:7800ff1208259bdf93c95b0ede350cabb5acc29c6a30f64bb4a588ac44348ae1 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (3): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Exception received when sending HTTP request. 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.444 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.445 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.446 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.447 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: retry needed, retryable exception caught: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.448 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.449 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.450 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.451 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 308, in _should_retry 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 364, in __call__ 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 248, in __call__ 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, caught_exception 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: raise caught_exception 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.452 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: Retry needed, action of: 3.0403270765421264 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Response received to retry, sleeping for 3.0403270765421264 seconds 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-4 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.453 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-acl:public-read-write 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T160301Z 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T160301Z 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/us-east-1/s3/aws4_request 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:38317800d34126783f6713715cd859970225ca7f8dfaa0e4f46e9e058af84167 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:785dd39c0b800837fd2ef4d333cbbcbe73e309e55d1495bac7f6d4f25bdef789 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.454 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (4): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Exception received when sending HTTP request. 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.455 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.456 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.457 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.458 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: retry needed, retryable exception caught: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.459 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.460 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.461 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.462 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 308, in _should_retry 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 364, in __call__ 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, response, caught_exception 2024-04-20T16:10:03.463 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 248, in __call__ 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: attempt_number, caught_exception 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: raise caught_exception 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.retryhandler: DEBUG: Retry needed, action of: 7.970517771708298 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Response received to retry, sleeping for 7.970517771708298 seconds 2024-04-20T16:10:03.464 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-4 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:host:smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-acl:public-read-write 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.465 INFO:teuthology.orchestra.run.smithi071.stderr:x-amz-date:20240420T160309Z 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: StringToSign: 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:AWS4-HMAC-SHA256 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:20240420T160309Z 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:20240420/us-east-1/s3/aws4_request 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:550b641e81eebe63bad5da01d4a6516714da7b4085b8cf33f1527560e48fa526 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.auth: DEBUG: Signature: 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:fc9a497da8b1339ca76bb5a67209f716517b7b132967d01e7eb3a1343782944a 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-04-20T16:10:03.466 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (5): smithi071.front.sepia.ceph.com:80 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.endpoint: DEBUG: Exception received when sending HTTP request. 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: (self._dns_host, self.port), self.timeout, **extra_kw 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.467 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 452, in send 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=self._chunked(request.headers), 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 800, in urlopen 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: raise six.reraise(type(error), error, _stacktrace) 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: raise value 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: chunked=chunked, 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: conn.request(method, url, **httplib_request_kw) 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: super(HTTPConnection, self).request(method, url, body=body, headers=headers) 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.468 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 95, in _send_request 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: method, url, body, headers, *args, **kwargs 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 123, in _send_output 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/awsrequest.py", line 218, in send 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: return super().send(str) 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 205, in connect 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: conn = self._new_conn() 2024-04-20T16:10:03.469 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: self, "Failed to establish a new connection: %s" % e 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr:urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr:During handling of the above exception, another exception occurred: 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 278, in _do_get_response 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: http_response = self._send(request) 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/endpoint.py", line 374, in _send 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: return self.http_session.send(request) 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/botocore/httpsession.py", line 472, in send 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr: raise EndpointConnectionError(endpoint_url=request.url, error=e) 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://smithi071.front.sepia.ceph.com:80/lbboah-4" 2024-04-20T16:10:03.470 INFO:teuthology.orchestra.run.smithi071.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test multipart object upload on master 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2078, in test_ps_s3_multipart_on_master 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.471 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.472 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-5/ 2024-04-20T16:10:03.473 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-5/ 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-5/ 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.475 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.476 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.477 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1184.572437986 and callback=>; now=1154.572437986; delay=30.0 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.479 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1184.584083285 and callback=>; now=1154.584083285; delay=30.0 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.480 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:03:09 GMT 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-5/ 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ= 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:03:09 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:irZCT59Oln9j/WEEbvfyvCmZKDQ=', 'Content-Length': '0'} 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.481 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 10 heartbeat frames, sent 10, idle intervals 0 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1244.586425957 and callback=>; now=1179.586425957; delay=65 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 10 heartbeat frames, sent 10, idle intervals 0 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1244.600474405 and callback=>; now=1179.600474405; delay=65 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1214.573250792 and callback=>; now=1184.573250792; delay=30.0 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.482 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1214.584952319 and callback=>; now=1184.584952319; delay=30.0 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.483 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1244.574308518 and callback=>; now=1214.574308518; delay=30.0 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.484 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1244.586084241 and callback=>; now=1214.586084241; delay=30.0 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test s3 notification of metadata on master 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2169, in test_ps_s3_metadata_filter_on_master 2024-04-20T16:10:03.485 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.486 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.487 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-6/ 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-6/ 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-6/ 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.488 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.489 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.490 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.491 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.492 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1274.575234668 and callback=>; now=1244.575234668; delay=30.0 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 13 heartbeat frames, sent 13, idle intervals 0 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1309.586832356 and callback=>; now=1244.586832356; delay=65 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1274.587390334 and callback=>; now=1244.587390334; delay=30.0 2024-04-20T16:10:03.494 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 13 heartbeat frames, sent 13, idle intervals 0 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1309.600939056 and callback=>; now=1244.600939056; delay=65 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:04:33 GMT 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-6/ 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0= 2024-04-20T16:10:03.495 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:04:33 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:WVlBot5+U6sFQWhP7I0iUJ0Ddd0=', 'Content-Length': '0'} 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test s3 notification of metadata on master 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2273, in test_ps_s3_metadata_on_master 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.496 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.497 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-7/ 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-7/ 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.498 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-7/ 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.499 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.500 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.501 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.502 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.503 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1304.576194246 and callback=>; now=1274.576194246; delay=30.0 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1304.588196455 and callback=>; now=1274.588196455; delay=30.0 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.505 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:05:08 GMT 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-7/ 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU= 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:05:08 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Ao7qLk1NqRNvIQZ3cFTNM4ZCrrU=', 'Content-Length': '0'} 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.506 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1334.577142468 and callback=>; now=1304.577142468; delay=30.0 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.507 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1334.589117212 and callback=>; now=1304.589117212; delay=30.0 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 15 heartbeat frames, sent 15, idle intervals 0 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1374.587840071 and callback=>; now=1309.587840071; delay=65 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 15 heartbeat frames, sent 15, idle intervals 0 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1374.601690968 and callback=>; now=1309.601690968; delay=65 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.508 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test s3 notification of tags on master 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2367, in test_ps_s3_tags_on_master 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.509 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.510 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-8/ 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-8/ 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-8/ 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.511 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.512 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.514 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.515 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1364.578186112 and callback=>; now=1334.578186112; delay=30.0 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1364.590172776 and callback=>; now=1334.590172776; delay=30.0 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.518 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:15 GMT 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-8/ 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc= 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:15 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:Y2UDoOwlGYJ+6VzORdYeup7wQrc=', 'Content-Length': '0'} 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.519 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1394.57911389 and callback=>; now=1364.57911389; delay=30.0 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1394.591325923 and callback=>; now=1364.591325923; delay=30.0 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test s3 notification of object versions 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2478, in test_ps_s3_versioning_on_master 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.521 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.522 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-9/ 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-9/ 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-9/ 2024-04-20T16:10:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.526 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 17 heartbeat frames, sent 17, idle intervals 0 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1439.589865606 and callback=>; now=1374.589865606; delay=65 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 17 heartbeat frames, sent 17, idle intervals 0 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1439.601830119 and callback=>; now=1374.601830119; delay=65 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.527 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.528 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1424.580189468 and callback=>; now=1394.580189468; delay=30.0 2024-04-20T16:10:03.529 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1424.59215999 and callback=>; now=1394.59215999; delay=30.0 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.530 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:06:58 GMT 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-9/ 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA= 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:06:58 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:vybxAmAB7qoECPMX1wAvyFG9DeA=', 'Content-Length': '0'} 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test s3 notification of deletion markers on master 2024-04-20T16:10:03.531 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 2553, in test_ps_s3_versioned_deletion_on_master 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.532 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.533 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-10/ 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-10/ 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-10/ 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.535 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.536 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.537 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.538 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.539 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1454.581316004 and callback=>; now=1424.581316004; delay=30.0 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1454.59319508 and callback=>; now=1424.59319508; delay=30.0 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.540 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.541 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:07:42 GMT 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-10/ 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY= 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:07:42 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:m9PtMRujYOZq32hp0/2mVNhHENY=', 'Content-Length': '0'} 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 19 heartbeat frames, sent 19, idle intervals 0 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1504.596102925 and callback=>; now=1439.596102925; delay=65 2024-04-20T16:10:03.542 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 19 heartbeat frames, sent 19, idle intervals 0 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1504.607890842 and callback=>; now=1439.607890842; delay=65 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test pushing persistent notification amqp 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 3346, in test_ps_s3_persistent_notification_amqp 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: persistent_notification('amqp') 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 3244, in persistent_notification 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.543 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.544 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.545 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-11/ 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-11/ 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-11/ 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.546 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.547 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.548 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.549 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.550 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1484.582262576 and callback=>; now=1454.582262576; delay=30.0 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.551 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1484.594157321 and callback=>; now=1454.594157321; delay=30.0 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.552 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.553 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:08:12 GMT 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-11/ 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY= 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:08:12 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:EhC385iwxlSu2avigzgAZjcEpGY=', 'Content-Length': '0'} 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1514.583195359 and callback=>; now=1484.583195359; delay=30.0 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1514.59524635 and callback=>; now=1484.59524635; delay=30.0 2024-04-20T16:10:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:====================================================================== 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:ERROR: test pushing persistent notification of large notifications 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr:Traceback (most recent call last): 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: self.test(*self.arg) 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py", line 3370, in test_ps_s3_persistent_notification_large 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: bucket = conn.create_bucket(bucket_name) 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 619, in create_bucket 2024-04-20T16:10:03.556 INFO:teuthology.orchestra.run.smithi071.stderr: data=data) 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1071, in make_request 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: retry_handler=retry_handler) 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 1030, in _mexe 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: raise ex 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/lib64/python3.6/site-packages/boto/connection.py", line 943, in _mexe 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: request.body, request.headers) 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1273, in request 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_request(method, url, body, headers, encode_chunked) 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1319, in _send_request 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: self.endheaders(body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: self._send_output(message_body, encode_chunked=encode_chunked) 2024-04-20T16:10:03.557 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: self.send(msg) 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 982, in send 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: self.connect() 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/http/client.py", line 954, in connect 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: (self.host,self.port), self.timeout, self.source_address) 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: raise err 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr: sock.connect(sa) 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr:ConnectionRefusedError: [Errno 111] Connection refused 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr:-------------------- >> begin captured logging << -------------------- 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using access key provided by client. 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Using secret key provided by client. 2024-04-20T16:10:03.558 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: path=/lbboah-12/ 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: auth_path=/lbboah-12/ 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Method: PUT 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Path: /lbboah-12/ 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Data: 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Headers: {} 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Host: smithi071.front.sepia.ceph.com 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Port: 80 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Params: {} 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.559 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.560 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 21 heartbeat frames, sent 21, idle intervals 0 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1569.601855011 and callback=>; now=1504.601855011; delay=65 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received 21 heartbeat frames, sent 21, idle intervals 0 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1569.613440995 and callback=>; now=1504.613440995; delay=65 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1544.584432735 and callback=>; now=1514.584432735; delay=30.0 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1544.596286723 and callback=>; now=1514.596286723; delay=30.0 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Token: None 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: StringToSign: 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:PUT 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:Sat, 20 Apr 2024 16:09:14 GMT 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:/lbboah-12/ 2024-04-20T16:10:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Signature: 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI= 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.6.8 Linux/4.18.0-552.el8.x86_64', 'Date': 'Sat, 20 Apr 2024 16:09:14 GMT', 'Authorization': 'AWS RPTKEXNTSFXEXSLLARJY:lBfUFEQZerWUK8f+q11gpbx6oKI=', 'Content-Length': '0'} 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: encountered ConnectionRefusedError exception, reconnecting 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 80} 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(11, >) 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(11, _) updated handler RdWr 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1574.585117581 and callback=>; now=1544.585117581; delay=30.0 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(11) 2024-04-20T16:10:03.567 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(11) updated handler Rd 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Received heartbeat frame 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Recv would block on 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.heartbeat: DEBUG: Sending heartbeat frame 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.set_writer(15, >) 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: set_writer(15, _) updated handler RdWr 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned on writability watcher: 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.select_connection: DEBUG: call_later: added timeout with deadline=1574.597125543 and callback=>; now=1544.597125543; delay=30.0 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: SelectorIOServicesAdapter.remove_writer(15) 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.selector_ioloop_adapter: DEBUG: remove_writer(15) updated handler Rd 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:pika.adapters.utils.io_services_utils: DEBUG: Turned off writability watcher: 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:--------------------- >> end captured logging << --------------------- 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.568 INFO:teuthology.orchestra.run.smithi071.stderr:---------------------------------------------------------------------- 2024-04-20T16:10:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:Ran 12 tests in 705.237s 2024-04-20T16:10:03.569 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:10:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:FAILED (errors=11) 2024-04-20T16:10:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:10:03.570 INFO:teuthology.orchestra.run.smithi071.stdout:send: b'GET /lbboah-1?notification=lbboah-1_notif_4 HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS RPTKEXNTSFXEXSLLARJY:s+2wWSKD+62UcPjzoovaVOfqYRc=\r\nDate: Sat, 20 Apr 2024 15:58:18 +0000\r\nHost: smithi071.front.sepia.ceph.com:80\r\n\r\n' 2024-04-20T16:10:03.570 INFO:teuthology.orchestra.run.smithi071.stdout:Zero length chunk ignored 2024-04-20T16:10:03.570 INFO:teuthology.orchestra.run.smithi071.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-04-20T16:10:03.570 INFO:teuthology.orchestra.run.smithi071.stdout:header: x-amz-request-id: tx000006c1f2eeeb6095396-006623e61a-11a9-default 2024-04-20T16:10:03.570 INFO:teuthology.orchestra.run.smithi071.stdout:header: Content-Type: application/xml 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: Content-Length: 504 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: Date: Sat, 20 Apr 2024 15:58:18 GMT 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: Connection: Keep-Alive 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:wait for 5sec for the messages... 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:send: b'DELETE /lbboah-1?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS RPTKEXNTSFXEXSLLARJY:kjAueLfDcKVGF+vULjsZyaqh9Jo=\r\nDate: Sat, 20 Apr 2024 15:58:26 +0000\r\nHost: smithi071.front.sepia.ceph.com:80\r\n\r\n' 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:Zero length chunk ignored 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: x-amz-request-id: tx00000e001a34197076d57-006623e622-11a9-default 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: Content-Length: 0 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: Date: Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:header: Connection: Keep-Alive 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:send: b'DELETE /lbboah-1?notification HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAuthorization: AWS RPTKEXNTSFXEXSLLARJY:G7q2n6RRhXHk3iNkej9xL/5n0nE=\r\nDate: Sat, 20 Apr 2024 15:58:27 +0000\r\nHost: smithi071.front.sepia.ceph.com:80\r\n\r\n' 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:Zero length chunk ignored 2024-04-20T16:10:03.571 INFO:teuthology.orchestra.run.smithi071.stdout:reply: 'HTTP/1.1 200 OK\r\n' 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:header: x-amz-request-id: tx00000aba2d0fcee049d60-006623e623-11a9-default 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:header: Content-Length: 0 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:header: Date: Sat, 20 Apr 2024 15:58:27 GMT 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:header: Connection: Keep-Alive 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.572 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:Error: [Errno 111] Connection refused 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:average time for creation + qmqp notification is: 970.1333522796631 milliseconds 2024-04-20T16:10:03.573 INFO:teuthology.orchestra.run.smithi071.stdout:wait for 5sec for the messages... 2024-04-20T16:10:03.574 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 238, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (bucket notification tests against different endpoints) on smithi071 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test' 2024-04-20T16:10:03.580 INFO:tasks.notification_tests:Removing bn-tests.conf file... 2024-04-20T16:10:03.580 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf 2024-04-20T16:10:03.600 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2024-04-20T16:10:03.684 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setuser ceph since I am not root 2024-04-20T16:10:03.684 INFO:teuthology.orchestra.run.smithi071.stderr:ignoring --setgroup ceph since I am not root 2024-04-20T16:15:03.686 INFO:teuthology.orchestra.run.smithi071.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-04-20T16:15:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:15:03.688 INFO:tasks.notification_tests:Pre-processing completed... 2024-04-20T16:15:03.688 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/.aws/models/s3/2006-03-01/service-2.sdk-extras.json 2024-04-20T16:15:03.751 DEBUG:teuthology.orchestra.run.smithi071:> cd /home/ubuntu/ && rmdir -p .aws/models/s3/2006-03-01/ 2024-04-20T16:15:03.818 INFO:tasks.notification_tests:Removing bucket-notifications-tests... 2024-04-20T16:15:03.818 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/ceph 2024-04-20T16:15:04.172 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 151, in create_users yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 238, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (bucket notification tests against different endpoints) on smithi071 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 306, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 45, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 158, in create_users ctx.cluster.only(client).run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' 2024-04-20T16:15:04.388 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=372804916c6c448788f312e6c191a6ec Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 151, in create_users yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 238, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (bucket notification tests against different endpoints) on smithi071 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a amqp_test' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 306, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 45, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/notification_tests.py", line 158, in create_users ctx.cluster.only(client).run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' 2024-04-20T16:15:04.393 DEBUG:teuthology.run_tasks:Unwinding manager notification-tests 2024-04-20T16:15:04.402 DEBUG:teuthology.run_tasks:Unwinding manager rabbitmq 2024-04-20T16:15:04.410 INFO:tasks.rabbitmq:Stopping RabbitMQ Service... 2024-04-20T16:15:04.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/service rabbitmq-server stop 2024-04-20T16:15:04.455 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl stop rabbitmq-server.service 2024-04-20T16:15:07.052 INFO:tasks.rabbitmq:Removing packaged dependencies of RabbitMQ... 2024-04-20T16:15:07.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y remove rabbitmq-server.noarch 2024-04-20T16:15:07.587 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-20T16:15:07.588 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout:Removing: 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: rabbitmq-server noarch 3.13.1-1.el8 @rabbitmq_rabbitmq-server 23 M 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-04-20T16:15:07.589 INFO:teuthology.orchestra.run.smithi071.stdout:Remove 1 Package 2024-04-20T16:15:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T16:15:07.590 INFO:teuthology.orchestra.run.smithi071.stdout:Freed space: 23 M 2024-04-20T16:15:07.590 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-20T16:15:07.594 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-20T16:15:07.594 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-20T16:15:07.621 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-20T16:15:07.622 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-20T16:15:07.679 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-20T16:15:07.766 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T16:15:07.767 INFO:teuthology.orchestra.run.smithi071.stdout: Erasing : rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T16:15:08.299 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T16:15:08.651 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rabbitmq-server-3.13.1-1.el8.noarch 1/1 2024-04-20T16:15:08.652 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T16:15:08.652 INFO:teuthology.orchestra.run.smithi071.stdout:Removed: 2024-04-20T16:15:08.652 INFO:teuthology.orchestra.run.smithi071.stdout: rabbitmq-server-3.13.1-1.el8.noarch 2024-04-20T16:15:08.652 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-20T16:15:08.652 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-20T16:15:08.871 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-20T16:15:08.882 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-04-20T16:15:08.882 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:15:08.882 INFO:tasks.rgw.client.0:Stopped 2024-04-20T16:15:08.883 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-04-20T16:15:08.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/vault-root-token 2024-04-20T16:15:08.969 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2024-04-20T16:15:08.978 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-20T16:15:08.988 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-04-20T16:15:08.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T16:17:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-20T16:17:09.042 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph.py", line 1904, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2323, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2557, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1640, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1631, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-20T16:17:09.053 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-20T16:17:09.053 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-20T16:17:09.053 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-20T16:17:09.053 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.056 INFO:tasks.ceph.osd.0:Stopped 2024-04-20T16:17:09.056 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-20T16:17:09.056 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.056 INFO:tasks.ceph.osd.1:Stopped 2024-04-20T16:17:09.057 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-20T16:17:09.057 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.057 INFO:tasks.ceph.osd.2:Stopped 2024-04-20T16:17:09.057 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-20T16:17:09.057 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.057 INFO:tasks.ceph.osd.3:Stopped 2024-04-20T16:17:09.057 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-20T16:17:09.058 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.058 INFO:tasks.ceph.osd.4:Stopped 2024-04-20T16:17:09.058 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-20T16:17:09.058 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.058 INFO:tasks.ceph.osd.5:Stopped 2024-04-20T16:17:09.058 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-20T16:17:09.059 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.059 INFO:tasks.ceph.osd.6:Stopped 2024-04-20T16:17:09.059 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-20T16:17:09.059 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.059 INFO:tasks.ceph.osd.7:Stopped 2024-04-20T16:17:09.059 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-20T16:17:09.059 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-20T16:17:09.060 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.060 INFO:tasks.ceph.mgr.y:Stopped 2024-04-20T16:17:09.060 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-20T16:17:09.060 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:17:09.060 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph.py", line 1444, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph.py", line 1904, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2323, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2557, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1640, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1631, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2024-04-20T16:17:09.070 INFO:tasks.ceph.mgr.x:Stopped 2024-04-20T16:17:09.070 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-20T16:17:09.071 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-20T16:17:09.071 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.071 INFO:tasks.ceph.mon.a:Stopped 2024-04-20T16:17:09.071 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-20T16:17:09.071 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.071 INFO:tasks.ceph.mon.c:Stopped 2024-04-20T16:17:09.071 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-20T16:17:09.072 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T16:17:09.072 INFO:tasks.ceph.mon.b:Stopped 2024-04-20T16:17:09.072 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-20T16:17:09.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-20T16:17:09.112 INFO:teuthology.orchestra.run.smithi071.stdout:1713628925.3305292 mon.a (mon.0) 272 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH) 2024-04-20T16:17:09.112 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-20T16:17:09.113 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T16:17:09.113 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-20T16:17:09.279 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T16:17:09.280 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-20T16:17:09.357 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T16:17:09.357 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-20T16:17:09.437 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T16:17:09.437 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-20T16:17:09.510 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T16:17:09.511 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-20T16:17:09.607 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T16:17:09.608 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-20T16:17:09.681 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T16:17:09.681 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-20T16:17:09.743 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T16:17:09.743 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-20T16:17:09.817 INFO:tasks.ceph:Archiving mon data... 2024-04-20T16:17:09.818 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/data/mon.a.tgz 2024-04-20T16:17:09.818 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-20T16:17:09.837 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.nEXhUsNfdv 2024-04-20T16:17:09.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.nEXhUsNfdv 2024-04-20T16:17:09.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.nEXhUsNfdv 2024-04-20T16:17:10.080 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.nEXhUsNfdv is 346KB 2024-04-20T16:17:10.177 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.nEXhUsNfdv 2024-04-20T16:17:10.194 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/data/mon.c.tgz 2024-04-20T16:17:10.194 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-20T16:17:10.250 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.yRf3pm2XBh 2024-04-20T16:17:10.251 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yRf3pm2XBh 2024-04-20T16:17:10.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.yRf3pm2XBh 2024-04-20T16:17:10.507 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.yRf3pm2XBh is 469KB 2024-04-20T16:17:10.605 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.yRf3pm2XBh 2024-04-20T16:17:10.622 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/data/mon.b.tgz 2024-04-20T16:17:10.622 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-04-20T16:17:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.dS9yNH2GwG 2024-04-20T16:17:10.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.dS9yNH2GwG 2024-04-20T16:17:10.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.dS9yNH2GwG 2024-04-20T16:17:10.922 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.dS9yNH2GwG is 468KB 2024-04-20T16:17:11.025 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.dS9yNH2GwG 2024-04-20T16:17:11.042 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-20T16:17:11.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T16:17:11.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T16:17:11.109 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-20T16:17:11.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-20T16:17:11.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-20T16:17:11.138 INFO:teuthology.orchestra.run.smithi071.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-20T16:17:11.177 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-20T16:17:11.182 INFO:tasks.ceph:Archiving crash dumps... 2024-04-20T16:17:11.184 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/remote/smithi071/crash 2024-04-20T16:17:11.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-20T16:17:11.244 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/remote/smithi119/crash 2024-04-20T16:17:11.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-20T16:17:11.272 INFO:tasks.ceph:Compressing logs... 2024-04-20T16:17:11.272 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T16:17:11.274 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T16:17:11.307 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50144.log 2024-04-20T16:17:11.307 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50375.log 2024-04-20T16:17:11.307 INFO:teuthology.orchestra.run.smithi071.stderr:gzip /var/log/ceph/ceph-client.admin.50144.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56410.log 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50144.log.gz 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50729.log 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50375.log.gz 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr:gzip /var/log/ceph/ceph-client.admin.56410.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50046.log 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56410.log.gz 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50729.log: gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50729.log.gz 2024-04-20T16:17:11.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50046.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50046.log.gz 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51271.log 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51487.log 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51271.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51271.log.gz 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.51487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50737.log 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51029.log 2024-04-20T16:17:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50737.log.gz 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51029.log.gz 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.0.51487.log.gz 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.smithi071.55643.log 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55884.log 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50369.log 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.crash.smithi071.55643.log: 52.2% -- replaced with /var/log/ceph/ceph-client.crash.smithi071.55643.log.gz 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.55884.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-20T16:17:11.310 INFO:teuthology.orchestra.run.smithi071.stderr: 6.1% -- replaced with /var/log/ceph/ceph-client.0.55884.log.gz 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49879.log 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50369.log.gz 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51367.log 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49879.log.gz 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51367.log.gz 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51463.log 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50688.log 2024-04-20T16:17:11.311 INFO:teuthology.orchestra.run.smithi071.stderr:gzip /var/log/ceph/ceph-client.admin.51463.log: -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51463.log.gz 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50768.log 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50688.log.gz 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49997.log 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50768.log.gz 2024-04-20T16:17:11.312 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.49997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49997.log.gz 2024-04-20T16:17:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49948.log 2024-04-20T16:17:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50373.log 2024-04-20T16:17:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49948.log.gz 2024-04-20T16:17:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50373.log.gz 2024-04-20T16:17:11.313 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51222.log 2024-04-20T16:17:11.321 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50370.log 2024-04-20T16:17:11.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51222.log.gz 2024-04-20T16:17:11.321 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50370.log.gz 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50372.log 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51173.log 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50372.log.gz 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51173.log.gz 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51414.log 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51028.log 2024-04-20T16:17:11.322 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51414.log.gz 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37420.log 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51028.log.gz 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.tmp-client.admin.37420.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37420.log.gz 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51318.log 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51594.log 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51318.log.gz 2024-04-20T16:17:11.323 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.51594.log: 76.7% -- replaced with /var/log/ceph/ceph-client.0.51594.log.gz 2024-04-20T16:17:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51770.log 2024-04-20T16:17:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50358.log 2024-04-20T16:17:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.51770.log: 76.5% -- replaced with /var/log/ceph/ceph-client.0.51770.log.gz 2024-04-20T16:17:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50358.log.gz 2024-04-20T16:17:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51682.log 2024-04-20T16:17:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55669.log 2024-04-20T16:17:11.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.51682.log: 76.8% -- replaced with /var/log/ceph/ceph-client.0.51682.log.gz 2024-04-20T16:17:11.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.55669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55669.log.gz 2024-04-20T16:17:11.325 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50374.log 2024-04-20T16:17:11.325 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.55656.log 2024-04-20T16:17:11.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50374.log.gz 2024-04-20T16:17:11.325 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.crash.55656.log: 52.5% -- replaced with /var/log/ceph/ceph-client.crash.55656.log.gz 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49772.log 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.49772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49772.log.gz 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50095.log 2024-04-20T16:17:11.326 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50095.log.gz 2024-04-20T16:17:11.327 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-20T16:17:11.328 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53668.log 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.53668.log: 84.2% -- replaced with /var/log/ceph/ceph-client.0.53668.log.gz 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44162.log 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44162.log.gz 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51858.log 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.0.51858.log: 76.7% -- replaced with /var/log/ceph/ceph-client.0.51858.log.gz 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50639.log 2024-04-20T16:17:11.329 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50639.log.gz 2024-04-20T16:17:11.330 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-20T16:17:11.330 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-20T16:17:11.330 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-04-20T16:17:11.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.44029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-04-20T16:17:11.331 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50727.log 2024-04-20T16:17:11.331 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50727.log.gz 2024-04-20T16:17:11.331 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51056.log 2024-04-20T16:17:11.332 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.51056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51056.log.gz 2024-04-20T16:17:11.332 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50977.log 2024-04-20T16:17:11.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50977.log.gz 2024-04-20T16:17:11.333 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50371.log 2024-04-20T16:17:11.334 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50371.log.gz 2024-04-20T16:17:11.334 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-20T16:17:11.342 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50739.log 2024-04-20T16:17:11.346 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43831.log 2024-04-20T16:17:11.346 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43536.log 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43831.log.gz 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43632.log 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43536.log.gz 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43681.log 2024-04-20T16:17:11.350 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43632.log.gz 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43584.log 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43681.log.gz 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43584.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 0.0% 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43584.log.gz 2024-04-20T16:17:11.351 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43782.log 2024-04-20T16:17:11.352 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-20T16:17:11.352 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.43782.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-20T16:17:11.352 INFO:teuthology.orchestra.run.smithi119.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43782.log.gz 2024-04-20T16:17:11.352 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-20T16:17:11.352 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43488.log 2024-04-20T16:17:11.353 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.43488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43488.log.gz 2024-04-20T16:17:11.353 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-20T16:17:11.353 INFO:teuthology.orchestra.run.smithi119.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-20T16:17:11.354 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.50739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50739.log.gz 2024-04-20T16:17:11.354 INFO:teuthology.orchestra.run.smithi071.stderr: 92.5% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2024-04-20T16:17:11.355 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43654.log 2024-04-20T16:17:11.356 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.43654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43654.log.gz 2024-04-20T16:17:11.356 INFO:teuthology.orchestra.run.smithi119.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-20T16:17:11.357 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43731.log 2024-04-20T16:17:11.358 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.43731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43731.log.gz 2024-04-20T16:17:11.379 INFO:teuthology.orchestra.run.smithi071.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-20T16:17:11.444 INFO:teuthology.orchestra.run.smithi119.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-20T16:17:11.449 INFO:teuthology.orchestra.run.smithi071.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-20T16:17:11.601 INFO:teuthology.orchestra.run.smithi071.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-20T16:17:11.710 INFO:teuthology.orchestra.run.smithi119.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-20T16:17:11.766 INFO:teuthology.orchestra.run.smithi071.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-20T16:17:11.824 INFO:teuthology.orchestra.run.smithi071.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-20T16:17:11.906 INFO:teuthology.orchestra.run.smithi119.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-20T16:17:11.933 INFO:teuthology.orchestra.run.smithi119.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-20T16:17:11.966 INFO:teuthology.orchestra.run.smithi071.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-20T16:17:11.997 INFO:teuthology.orchestra.run.smithi119.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-20T16:17:11.998 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-20T16:17:11.998 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m0.662s 2024-04-20T16:17:11.999 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m2.177s 2024-04-20T16:17:11.999 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m0.093s 2024-04-20T16:17:12.000 INFO:teuthology.orchestra.run.smithi071.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-20T16:17:12.000 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-20T16:17:12.000 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m0.711s 2024-04-20T16:17:12.000 INFO:teuthology.orchestra.run.smithi071.stderr:user 0m2.782s 2024-04-20T16:17:12.000 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m0.133s 2024-04-20T16:17:12.001 INFO:tasks.ceph:Archiving logs... 2024-04-20T16:17:12.001 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/remote/smithi071/log 2024-04-20T16:17:12.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T16:17:12.497 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/remote/smithi119/log 2024-04-20T16:17:12.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T16:17:12.870 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph.py", line 1904, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2323, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2557, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1640, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1631, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-20T16:17:12.880 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-20T16:17:12.893 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph.py", line 1904, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2323, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 2557, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1640, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_7f4fa80c8c97f53f52b1189d2347b5761f6287e3/qa/tasks/ceph_manager.py", line 1631, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-20T16:17:12.897 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-20T16:17:12.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T16:17:12.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T16:17:12.981 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-20T16:17:12.994 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-20T16:17:12.994 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T16:17:12.997 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T16:17:13.012 INFO:teuthology.orchestra.run.smithi071.stderr:bash: ntpq: command not found 2024-04-20T16:17:13.015 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2024-04-20T16:17:13.016 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T16:17:13.016 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-04-20T16:17:13.017 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 -180us[ -176us] +/- 81ms 2024-04-20T16:17:13.017 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -403us[ -399us] +/- 73ms 2024-04-20T16:17:13.017 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +632us[ +632us] +/- 88ms 2024-04-20T16:17:13.017 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:17:13.019 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T16:17:13.019 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-20T16:17:13.020 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 -171us[ -171us] +/- 81ms 2024-04-20T16:17:13.020 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -377us[ -356us] +/- 73ms 2024-04-20T16:17:13.020 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +626us[ +626us] +/- 88ms 2024-04-20T16:17:13.020 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:17:13.021 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-20T16:17:13.032 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-20T16:17:13.032 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-20T16:17:13.047 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T16:17:13.122 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T16:17:13.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T16:17:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:17:13.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T16:17:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:17:13.246 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-20T16:17:13.257 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-20T16:17:13.271 INFO:teuthology.task.internal:Duration was 1951.690022 seconds 2024-04-20T16:17:13.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-20T16:17:13.283 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-20T16:17:13.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T16:17:13.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T16:17:13.329 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T16:17:13.331 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T16:17:13.634 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-20T16:17:13.634 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-04-20T16:17:13.634 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T16:17:13.656 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-04-20T16:17:13.656 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T16:17:13.678 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-20T16:17:13.679 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T16:17:13.699 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T16:17:13.760 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-20T16:17:13.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T16:17:13.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T16:17:13.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-20T16:17:13.893 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-20T16:17:13.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T16:17:13.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T16:17:13.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-20T16:17:13.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T16:17:13.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T16:17:13.987 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-04-20T16:17:14.013 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-04-20T16:17:14.034 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T16:17:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:17:14.060 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T16:17:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:17:14.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-20T16:17:14.104 INFO:teuthology.task.internal:Transferring archived files... 2024-04-20T16:17:14.105 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/remote/smithi071 2024-04-20T16:17:14.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T16:17:14.144 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647/remote/smithi119 2024-04-20T16:17:14.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T16:17:14.182 INFO:teuthology.task.internal:Removing archive directory... 2024-04-20T16:17:14.185 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T16:17:14.187 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T16:17:14.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-20T16:17:14.249 INFO:teuthology.task.internal:Not uploading archives. 2024-04-20T16:17:14.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-20T16:17:14.263 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-20T16:17:14.263 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T16:17:14.265 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T16:17:14.280 INFO:teuthology.orchestra.run.smithi071.stdout: 526076 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 16:17 /home/ubuntu/cephtest 2024-04-20T16:17:14.292 INFO:teuthology.orchestra.run.smithi119.stdout: 526037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 16:17 /home/ubuntu/cephtest 2024-04-20T16:17:14.293 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-20T16:17:14.304 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-20T16:17:14.320 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-20T16:17:14.320 INFO:teuthology.nuke:Checking targets against current locks 2024-04-20T16:17:14.354 DEBUG:teuthology.nuke:shortname: smithi071 2024-04-20T16:17:14.354 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T16:17:14.377 DEBUG:teuthology.nuke:shortname: smithi119 2024-04-20T16:17:14.377 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T16:17:14.381 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 15:32:48.014844', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDb2dPrGNonFiPBD42oeSD+ltZCSt7KxYjqQLgxirOxBqzNetAzxLREfOYmzF4MpHrzNQitBAOmQXxMU0UXxK+E='} 2024-04-20T16:17:14.405 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:31:09-rgw-wip-yuri4-testing-2024-04-19-0708-quincy-distro-default-smithi/7664647', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 15:32:48.013482', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJr7BmfmWrdB1CO79XiqR7yDTi5evHE7CHFwOfOPXCu5jkFmr+QPbFz8h5Sn5L62sRrU2MLudH+ncq8K5mrgzA='} 2024-04-20T16:17:14.432 INFO:teuthology.orchestra.console.smithi071:Power off 2024-04-20T16:17:14.433 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T16:17:14.461 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2024-04-20T16:17:14.462 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:14.486 INFO:teuthology.orchestra.console.smithi119:Power off 2024-04-20T16:17:14.486 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T16:17:14.502 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-04-20T16:17:14.517 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-04-20T16:17:14.517 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:14.543 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-20T16:17:18.503 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:18.544 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:18.619 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-04-20T16:17:18.657 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-20T16:17:22.622 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:22.658 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:22.733 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-04-20T16:17:22.769 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-20T16:17:26.738 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:26.770 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T16:17:26.849 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2024-04-20T16:17:26.881 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-04-20T16:17:26.951 INFO:teuthology.orchestra.console.smithi071:Power off completed 2024-04-20T16:17:26.983 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-04-20T16:17:27.088 INFO:teuthology.run:Summary data: description: rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} duration: 1951.6900217533112 failure_reason: 'Command failed on smithi071 with status 1: ''adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=372804916c6c448788f312e6c191a6ec status: fail success: false 2024-04-20T16:17:27.088 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T16:17:27.161 INFO:teuthology.run:FAIL